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ARI  Research  Reports  and  Technical  Reports  are  intended  for  sponsors  of 
R&D  tasks  and  for  other  research  and  military  agencies.  Any  findings  ready 
for  implementation  at  the  time  of  publication  are  presented  in  the  last  part 
of  the  Brief.  Upon  completion  of  a  major  phase  of  the  task,  formal  recom¬ 
mendations  for  official  action  normally  are  conveyed  to  appropriate  military 
agencies  by  briefing  or  Disposition  Form. 


FOREWORD 


The  research  reported  here  is  part  of  a  broader  program  on  air  defense 
weapon  systems  being  conducted  by  the  Army  Research  Institute  Field  Unit  at 
Fort  Bliss,  Tex.  Since  1975,  the  Fort  Bliss  Field  Unit  has  been  engaged  in  a 
vigorous  program  of  research  to  discover  the  methods,  procedures  and  data  avail¬ 
able  for  conducting  cost  and  training  effectiveness  analyses  (CTEA) .  An  espe¬ 
cially  challenging  issue  is  the  development  of  a  methodology  applicable  to  wea¬ 
pon  systems  still  in  the  formative  or  conceptual  stage  of  development. 

The  methodology  presented  in  this  report  begins  with  preliminary  duty  or 
task  description  lists  that  include  probable  weapon  characteristics.  The  meth¬ 
odology  then  utilizes  a  series  of  selection  procedures  to  generate  a  profile  of 
equipment  and  methods  that  could  be  used  to  train  the  tasks.  The  selection 
procedures  also  yield  an  efficiency  ratio  that  indicates  the  relative  opportu¬ 
nity  costs  of  making  alternate  training  decisions  and  assumptions.  The  final 
projected  training  requirements  are  formulated  to  be  costed  in  dollars  via  a 
computer  program. 

While  the  methodology  was  developed  with  a  view  toward  application  on  the 
many  new  air  defense  systems  projected  for  fielding  in  the  mid-80's,  it  is  by 
no  means  restricted  to  air  defense  application.  The  methodology  is  intended 
for  use  in  any  man-machine  system  for  which  early  estimations  of  cost  and  train¬ 
ing  effectiveness  are  required. 

The  entire  program  is  responsive  to  the  requirements  of  RDTE  Project 
2Q76374 3A771  and  the  Director  of  Training  Developments,  U.S.  Army  Air  Defense 
School,  Fort  Bliss,  Tex. 


'JOSEPH  gEJDNER 
Technical  Director 
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EARLY  FORMULATION  OF  TRAINING  PROGRAMS 
FOR  COST  EFFECTIVENESS  ANALYSIS 


BRIEF 


Increased  accuracy  in  early  budgetary  prediction  has  become  more  important 
in  recent  years  and  has  led  to  a  corresponding  need  within  all  branches  of  the 
military  services  to  better  estimate  future  training  programs  and  their  impacts 
Recent  examples  of  this  need  have  included  the  interservice  procedures  for  in¬ 
structional  system  development1  (ISD) ,  TRADOC  Pamphlet  71-10, 2  and  the  Army 
Training  Study.3 

The  need  to  assess  the  value  of  training  programs  and  weapon  systems  has 
also  led  to  the  development  of  cost  and  operational  effectiveness  analysis 
(COEA)  and  associated  analyses,  such  as  cost  and  training  effectiveness  analy¬ 
sis  (CTEA)  and  reliability,  availability,  and  maintainability  analysis  (RAM) . 
Each  decision  making  tool,  however,  assumes  a  methodology  by  which  the  required 
information  can  be  generated.  In  CTEA  the  methodologies  are  still  in  the  de¬ 
velopmental  stages  as  a  result  of  both  instructional  decision  making  and  the 
current  state  of  the  art  in  training  analysis.  This  paper  addresses  the  latter 
area.  Its  concern  is  with  developing  a  user-oriented  procedure  for  early  for¬ 
mulation  of  training  programs  that  can  be  input  into  cost  and  training  ef¬ 
fectiveness  analyses. 

The  research  reported  in  this  paper  begins  with  a  theoretical  discussion 
of  the  procedure  and  proceeds  to  the  specific.  The  first  section  (Section  A) 
deals  mainly  with  the  underlying  logic  used  in  constructing  the  procedure.  The 
discussion  becomes  more  applied  in  Section  B.  Finally,  in  Section  C  a  simpli¬ 
fied  example  is  presented  designed  to  show  how  the  procedure  would  function  to 
choose  a  sample  training  method  and  its  support  equipment. 

An  overview  of  the  procedure  is  shown  in  the  flowchart  in  Figure  1.  Each 
box  in  the  flowchart  corresponds  to  a  key  process  that  must  be  performed  to 
formulate  a  future  training  program.  Four  of  these  processes:  Choose  media, 
record  efficiency,  describe  functional  context,  and  select  a  group  method  are 
considered  in  detail.  The  remaining  boxes  are  discussed  only  in  enough  detail 
to  illustrate  how  the  processes  work  together  for  input  into  CTEA. 

The  starting  point  for  this  procedure  is  a  task  description.  Much  of  the 
initial  task  analysis  information  for  developing  weapon  systems  is  not  avail¬ 
able  in  the  suggested  Instructional  Systems  Development  (ISD)  Handbook  format. 
Thus,  the  method  in  this  paper  has  not  been  made  dependent  upon  any  particular 
task  format,  but  instead  recodes  task  information  into  a  set  of  descriptive 
variables.  CTEA  processes  are,  therefore,  defined  in  terms  of  interactions 
among  these  variables.  Four  separate  variable  matrices  have  been  generated  to 
select  training  media  and  methods.  Each  matrix  corresponds  to  some  function  of 
a  training  program.  The  four  matrices  include:  (1)  The  stimulus  media  (equip¬ 
ment/materials)  prese  iting  the  training  information,  (2)  the  response  media 
(equipment/materials)  accepting  the  responses  made  by  the  student  to  the 
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stimulus  information,  (3)  the  feedback  media  (equipment/materials)  performing 
evaluations  of  the  student  response,  and  (4)  the  method,  i.e.,  the  organiza¬ 
tional  structure  of  student-instructor  interrelationships,  within  which  the 
media  are  used. 


Figure  1.  System  formulation. 


In  the  three  media  matrices,  physical  information  derived  from  each  task 
description  is  used  to  select  an  "ideal"  piece  of  training  equipment.  This 
physical  information  is  derived  not  only  from  the  task  description  but  also 
from  the  referenced  characteristics  of  the  system  hardware  upon  which  the  task 
is  performed. 

Unlike  the  media  matrices  where  physical  information  and  system  hardware 
derive  the  selection  of  stimulus,  response,  and  feedback  equipment,  the  training 
method  matrix  is  based  on  variables  which  describe  operational  context.  Because 
most  of  the  commonly  used  methods  are  loosely  defined,  a  separate  procedure  is 
used  to  more  precisely  define  alternative  training  approaches. 
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Although  the  method  and  media  variables  aid  training  program  selection  de¬ 
cisions,  a  technique  is  also  required  to  monitor  effects  produced  by  managerial 
changes.  This  paper  presents  a  process  called  "efficiency  ratio  calculation" 
to  track  the  psychological  training  gains  or  losses  caused  by  program  modifica¬ 
tions.  A  costing  process  to  track  dollar  values  is  also  considered.  Both  mea¬ 
sures  comprise  a  cost/efficiency  ratio  which  can  be  used  by  the  CTEA  analyst  to 
compare  the  relative  merits  of  various  training  programs . 
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EARLY  FORMULATION  OF  TRAINING  PROGRAMS 
FOR  COST  EFFECTIVENESS  ANALYSIS 


I.  TECHNICAL  SUPPLEMENT 

A.  THE  LOGIC  OF  GOING  FROM  TASK  DESCRIPTION  TO  TRAINING  PROGRAM 


To  go  from  a  task  description  to  a  costed  training  program,  a  procedure 
needs  to  be  set  up  for  defining  the  output  {the  training  program  elements)  in 
terms  of  a  series  of  manipulations  on  the  input  (task  analysis  data) .  This 
procedure,  or  mapping,  involves  finding  a  common  set  of  variables  by  which  both 
the  training  program  and  the  task  analysis  can  be  described.  (See  Fig.  2.) 

Once  a  common  set  of  variables  is  found,  it  is  then  possible  to  determine  how 
well  a  particular  training  concept  represents  critical  task  information  by  com¬ 
paring  similarities  or  differences  for  each  variable. 

Implicit  in  the  use  of  such  variables  are  some  distinct  methodological  as¬ 
sumptions  which  should  be  examined.  Most  training  experts  agree  that  two  im¬ 
portant  factors  determining  the  success  of  a  training  program  are  functional 
and  physical  fidelity.  Physical  fidelity  means  that  the  training  program  must 
capture  the  essential  operations  and  physical  conditions  present  in  the  real 
life  situation  for  which  an  individual  is  being  trained.  In  the  case  of  func¬ 
tional  fidelity,  the  critical  actions  or  functions  performed  by  the  system  user 
are  captured  by  either  exact  replication  of  user  actions  or  by  using  a  psycho¬ 
logically  useful  analog.  Specific  rules  for  generating  such  an  analog  are  not 
yet  (in  the  opinion  of  the  authors)  sufficiently  developed  to  be  useful  in  CTEA 
application.  The  importance  of  both  types  of  fidelity  depends  upon  a  wide  va¬ 
riety  of  factors  in  the  learning  situation. 

It  is  obvious  that  one  way  to  maximize  physical  and  functional  fidelity  is 
to  do  all  training  on  the  actual  system  equipment  and  in  the  real  environment. 
Allowing  for  the  fact  that  personnel  must  still  be  taught  certain  prerequisite 
skills  and  knowledges  before  they  can  begin  to  use  the  actual  equipment,  this 
approach  is  both  highly  expensive  and  unrealistic.  Consequently,  the  problem 
becomes  one  of  choosing  some  subset  of  the  full  range  of  physical  and  functional 
requirements,  combining  them  with  groups  of  devices  and  methods,  and  managing 
the  interactions  that  are  generated  as  a  result. 

There  are  at  least  two  approaches  for  determining  subsets.  One  approach 
is  to  select  important  fidelity  related  variables  by  analyzing  the  current  lit¬ 
erature  on  training  interactions.  One  difficulty  in  assimilating  this  litera¬ 
ture,  however,  is  that  the  contexts  within  which  studies  take  place  are  so  var¬ 
ied  that  it  is  often  difficult  to  identify  causal  relations.  Good  summaries 
have  been  produced,  e.g.,  Meister,  1976, 4  however,  and  are  a  great  aid  in  pull¬ 
ing  together  this  diversified  literature. 

A  second  approach  is  to  use  survey  information  drawn  from  panels  of  ex¬ 
perts.  This  type  of  information  is  generally  more  relevant  for  specific  appli¬ 
cations,  but  it  often  lacks  experimental  rigor  and  is  difficult  to  validate. 

If  forced  to  choose,  summary  literature  seems  to  present  the  more  immediate 
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Figure  2.  Mapping  from  task  situations  to  training  programs. 


data  base  for  preliminary  models  (Haver land,  1974^) .  The  model  developed  in 
this  paper  has  utilized  summary  literature  to  a  large  extent. 

A. 1.  A  STRING  ANALOGY 


The  model  assumes  an  underlying  logic  about  how  the  interactions  and  proc¬ 
esses  in  training  programs  can  be  represented.  The  lOgire— cgin  be  visually  il¬ 
lustrated  by  a  string  of  letters,  for  example:  ABCCBFCCBA.  Within  any  string, 
several  things  may  vary.  First,  the  string  is  composed  of  elements,  represented 
here  by  capital  letters.  Second,  the  elements  may  occupy  different  positions 
and  occur  at  different  frequencies  within  the  string.  Third,  the  string  may 
vary  in  length.  Finally  it  may  not  contain  all  possible  elements  that  could 
have  been  chosen.  A  training  program  can  be  similarly  represented. 

In  a  training  program  there  are  two  basic  levels  of  interest,  the  program 
level  and  the  content  level.  The  program  level  is  concerned  with  the  physical 
equipment  and  facilities  used  in  the  training  environment  and  the  social  struc¬ 
ture  or  method  within  which  the  physical  equipment  is  applied.  The  content 
level  is  concerned  with  the  units  of  training  information  which  are  flowing 
through  the  program  to  a  student  and  are  important  to  a  course  writer.  The  in¬ 
terrelationships  between  information  units  represent  structure,  which  when  spe¬ 
cified,  create  a  course  text.  Both  levels  of  interest  are  presented  below. 


Program  level 

Content  level 

Equipment  &  facilities 

Information  units 

(Media) 

(Content) 

Social  interrelationships 

Information  unit  interrelationships 

(Method) 

(Structure) 

Both  the  program  level  and  the  content  level  can  be  thought  of  as  a  string 
of  elements,  as  described  above.  For  the  program  level,  the  elements  are  the 
physical  media.  The  interrelationships  represented  in  the  ordering  of  the 
string  are  the  methods.  For  the  content  level,  the  elements  are  the  units  of 
information  to  be  presented  and  the  interrelationships  are  the  orderings  and 
sequences  of  information  units . 

As  with  the  string  analogy,  elements  at  the  program  level  may  occur  at 
different  frequencies,  may  be  selected  from  a  larger  pool,  may  be  grouped  into 
strings  of  different  lengths,  and  may  occupy  different  positions.  Elements  at 
the  content  level  may  themselves  be  composed  of  smaller  elements,  occur  at  dif¬ 
ferent  frequencies  of  presentation,  and  so  on. 
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Other  characteristics  of  a  string  concern  relational  properties  which  the 
elements  have  to  one  another.  For  example,  at  the  program  level,  the  concern 
is  how  the  media  interrelate.  This  interrelationship  is  called  the  training 
method.  Using  the  string  as  an  analogy,  a  convenient  breakdown  for  training 
progams  is  made  possible.  The  program  breakdown  in  turn  leads  to  analytical 
representations  which  can  be  paired  to  training  variables.  For  example,  five 
questions  were  derived,  using  the  string  analogy,  to  capture  the  basic  differ¬ 
ences  between  existing  training  methods.  First,  if  a  string  of  training  ele¬ 
ments  does  not  have  a  constant  sequence  over  time,  who  or  what  controls  the 
sequence?  Second,  how  many  times  in  an  instructional  string  does  an  element 
appear?  Third,  where  does  each  appearance  occur?  Fourth,  if  an  element  influ¬ 
ences  the  sequence  of  other  elements ,  what  degree  of  influence  does  it  have? 

For  example,  if  the  elements  were  an  instructor,  a  student  teacher,  and  stu¬ 
dents,  what  would  be  the  degree  of  control  of  the  student  teacher  relative  to 
the  instructor  in  a  given  method?  Finally,  what  percentage  of  the  total  in¬ 
structional  time  is  taken  up  by  each  of  the  elements? 

These  five  questions  were  systematically  applied  to  a  number  of  commonly 
accepted  training  "methods"  to  give  them  a  more  precise  definition  for  CTEA 
usage.  Twenty-three  such  methods  are  defined  in  Appendix  B-54-76,  along  with 
their  common  narrative  descriptions.  This  procedure  not  only  allows  one  to 
describe  an  existing  method  but  also  allows  one  to  produce  a  number  of  new 
training  methods  based  on  combinations  of  answers  to  the  five  basic  questions. 

Until  now  the  string  analogy  has  been  used  only  to  consider  the  program 
level,  but  it  is  equally  applicable  for  the  content  level.  For  this  level,  the 
elements  are  units  of  specific  training  material,  i.e.,  blocks  of  course  con¬ 
tent  at  any  convenient  level  of  description,  and  their  interrelationships,  the 
variables  affecting  the  structure,  i.e.,  the  sequencing  or  patterning  of  mate¬ 
rial.  At  this  level  of  analysis,  the  problem  is  one  of  specific  course  mate¬ 
rial  and  structure,  rather  than  the  determination  of  physical  equipment  and 
method.  In  this  way  the  string  can  be  used  as  a  convenient  framework  for  the 
division  and  clarification  of  processes  going  on  in  a  training  program. 

It  should  be  noted  that  a  training  program  does  not  operate  in  a  vacuum 
and  that  its  success  or  failure  is  influenced  markedly  by  its  environment. 
Consequently,  training  program  formulation  must  allow  for  external  impacts, 
such  as  managerial  limitations,  costs,  personnel,  and  so  on.  In  the  context  of 
a  string,  these  may  be  considered  modifying  factors  which  change  an  idealized 
string  of  instructional  events  to  a  conforming  string  concerned  with  practical 
realities.  In  this  paper,  such  modifying  factors  are  seen  as  having  a  direct 
impact  on  the  five  questions  used  to  define  the  training  methods. 

A. 2.  LINKING  THE  TASK  DATA  WITH  A  TRAINING  PROGRAM 

In  the  earlier  discussion  of  the  training  variables,  it  was  stated  that 
their  purpose  was  to  form  a  common  linkage  between  the  physical  and  functional 
properties  of  the  task  environment  and  the  potential  training  program.  The  way 
in  which  this  linkage  is  made  will  now  be  considered  in  greater  detail. 


Matrix  algebra  is  a  technique  that  can  be  used  in  the  application  of  both 
the  string  analogy  and  the  variables.  Because  of  its  flexibility,  matrix  alge¬ 
bra  is  a  powerful  tool  for  characterizing  any  model.  In  this  particular  effort, 
matrices  are  at  the  heart  of  the  method  and  media  selection  processes.  A  sam¬ 
ple  process  will  be  examined  to  show  how  matrix  techniques  are  used. 

Recall  that  the  variables  are  chosen  to  link  the  task  environment  to  some 
aspect  of  the  training  environment.  Figure  3  shows  a  sample  matrix.  Along  the 
X-axis  are  elements  representing  either  program  or  content  elements.  Along  the 
Y-axis  are  variables  representing  some  subset  of  training  variables  correspond¬ 
ing  to  a  particular  training  program  function  such  as  media  selection.  Each 
variable  along  the  Y-axis  is  given  a  value  based  on  its  applicability  or  non¬ 
applicability  to  each  program  or  content  element  along  the  X-axis.  For  exam¬ 
ple,  if  the  element  is  a  "slide  projector,"  and  a  variable  is  "visual  presenta¬ 
tion  capability,"  the  variable  would  be  applicable.  Using  the  same  logic,  each 
element  is  then  rated  in  terms  of  all  of  the  variables  for  the  training  program 
function.  The  value's  applicability  or  nonapplicability  is  recorded  in  the 
appropriate  cell  as  a  nominal,  ordinal,  or  ratio  value,  depending  upon  the  de¬ 
gree  of  accuracy  with  which  the  assessment  can  be  made.  The  complete  matrix 
thus  represents  a  description  of  the  set  of  training  elements  available  to  a 
training  manager  for  that  particular  function.  Such  matrices  are  constructed 
for  each  of  the  training  program  functions  for  which  training  literature  indi¬ 
cates  such  pairings  are  practical.  When  the  matrices  are  completed,  the  train¬ 
ing  elements  will  have  been  described  in  terms  of  variables. 

A  parallel  problem,  however,  is  how  is  task  information  described  in  terms 
of  the  same  variables?  A  similar  procedure  can  be  used.  For  example,  suppose 
that  a  task  involves  reading  visual  material  from  a  radar  scope.  If  the  task 
also  requires  a  response  to  some  kind  of  audio  information,  the  variables  ap¬ 
plicable  to  the  task  description  will  not  match  the  variables  applicable  to  the 
slide  projector  in  the  example  above.  It  might,  however,  match  some  other  piece 
of  equipment.  How  this  is  determined  is  where  a  matrix  becomes  valuable.  One 
difficult  way  in  which  a  comparison  can  be  performed  is  to  individually  examine 
each  variable  for  each  task  and  compare  it  to  each  variable  for  each  training 
element  in  a  process  matrix,  such  as  in  Figure  3.  Once  done,  the  number  of 
matching  variables  can  be  totaled  for  each  training  element  and  the  one  having 
the  highest  score  taken  as  the  "best  fit"  for  that  task.  If  the  procedure  is 
repeated  for  all  tasks  and  all  variables  in  each  of  the  training  processes  and 
are  combined  into  a  single  program,  a  training  analyst  has  a  starting  point 
from  which  to  do  predictive  analysis.  Needless  to  say,  this  procedure  would  be 
extremely  time  consuming,  if  done  entirely  by  hand.  Fortunately,  such  a  task 
is  made  easy  through  the  use  of  matrix  algebra  and  the  procedure  is  well  suited 
to  machine  implementation. 


A. 3.  DEVELOPING  THE  PROCESS  MATRICES 

Because  the  objective  in  the  previous  section  is  to  choose  the  device  or 
method  having  the  largest  number  of  similar  variables,  the  matrix  operation  of 
premultiplication  is  an  ideal  technique.  Suppose  that  the  task  description  is 
characterized  by  only  three  variables.  The  corresponding  matrix  cells  for 
these  variables  are  given  a  value  of  1  when  they  apply.  If  variables  do  not 
apply*  they  are  valued  as  0.  Similarly,  suppose  there  are  only  two  training 
elements  to  be  considered  in  the  process,  and  they  are  also  coded  with  a  1  or 
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Figure  3.  Linking  training  variables 
to  string  elements. 
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0.  (In  reality,  any  nominal,  ordinal,  or  ratio  value  could  be  used.)  To  see 
which  element  best  "fits"  the  above  task,  the  device  matrix  (A)  is  premultiplied 
by  the  task  variable  vector  (b) ,  producing  a  row  vector  of  column  sums  (c) . 

This  vector  represents  the  sum  of  the  number  of  variables  which  match  both  the 
task  description  and  the  device  matrix.  (See  Fig.  4.)  Therefore,  the  second 
column  element  in  the  matrix  is  viewed  as  the  "best  fit"  for  properties  in  the 
task  description,  i.e.,  it  matches  two,  the  most,  task  variables.  Suppose, 
however,  that  there  are  external  factors,  such  as  managerial  constraints,  af¬ 
fecting  the  suitability  of  elements  described  by  matrix  (A) .  If  so,  this  im¬ 
plies  that  the  values  in  (A)  should  be  weighted.  Use  of  matrix  formats  makes 
this  a  simple  operation.  For  example,  if  a  weight  is  given  to  each  element 
that  corresponds  to  its  relative  desirability,  then  each  column  of  variables  in 
the  matrix  must  be  multiplied  by  the  appropriate  weight.  Say  that  the  element 
in  column  one  is  four  times  as  suitable  as  the  element  in  column  two.  A  column 
multiplication  of  (A)  by  a  vector  of  weights,  w  =  (4,  1),  produces: 


Element 
#1  #2 


The  previous  premultiplication  by  (b)  now  produces  a  summed  vector  c  =  (4,  2) . 
Element  number  one  would  now  be  the  best  choice  considering  these  new  condi¬ 
tions.  The  same  logic  can  be  used  for  variables  in  the  task  description,  the 
importance  of  variables  in  the  matrix,  or  even  individual  items  in  the  matrix 
through  row  multiplication,  column  multiplication  or  matrix  element  replacement, 
respectively. 

Thus  the  use  of  matrix  algebra  presents  a  powerful  analytic  tool  for  ap¬ 
plying  a  wide  range  of  potential  interactions  and  factors  affecting  the  proc¬ 
esses  for  formulation  of  training  programs.  A  matrix  process  is  for  all  prac¬ 
tical  purposes  unlimited  in  terms  of  the  number  of  variables  it  can  consider, 
as  well  as  the  number  of  intervening  factors  that  can  modify  it.  In  these  sim- 
nle  examples,  only  two  or  three  elements  were  dealt  with.  In  the  computer  pro¬ 
gram  for  this  model,  program  elements  alone  number  more  than  35  for  a  single 
process.  The  logic,  however,  remains  the  same  in  both  cases. 


B.  APPLICATION  OF  THE  METHOD 


Up  to  this  point,  only  the  theoretical  framework  for  training  program  for¬ 
mulation  has  been  considered.  It  is  now  necessary  to  explore  the  specific  ways 
in  which  these  ideas  are  implemented.  First,  generation  of  the  training  vari¬ 
ables  will  be  considered,  since  they  are  central  for  developing  potential 
training  programs. 
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Step  1 


Step  2 


Step  3 


Step  4 


Recode  task  B  into  variable  row  vector  b. 

Let:  b  =  fb,  1,  l} 

Choose  a  device  matrix  such  as  in  Figure  2,  and  call  it  A. 

Elements 
1  2 


Variables 


3  0  1 


1  1  0 
2  1  1 


Pre-multiply  A  by  b  and  call  the  result  c. 

Let:  bA  =  {o,  1,  l)  p  ol  =  {l,  2} 


1  1 


Save  c  for  later  element  choices. 
Element 


fl  ■  2} 


Figure  4.  An  example  of  a  device  matrix  being  premultiplied 
by  a  task  variable  vector. 
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B.l.  GENERATING  THE  TRAINING  VARIABLES 


The  number  and  kinds  of  variables  affecting  training  programs  that  have 
been  considered  in  the  psychological  and  training  literature  is  extensive. 

Anyone  attempting  to  systematically  reduce  these  variables  into  a  usable  format 
is  struck  immediately  by  the  fact  that  in  each  experiment  and  taxonomy  variables 
are  confounded  in  at  least  two  ways.  First,  there  are  no  standard  variable  def¬ 
initions.  Second,  there  is  little  or  no  effort  made  to  assure  that  the  vari¬ 
ables  partition  the  space  of  training  alternatives  in  an  orthogonal,  or  at 
least,  non-overlapping  fashion.  Many  variables  given  different  names  have 
within  their  domain  the  same  variances  attributed  to  other  variables.  There 
are,  of  course,  exceptions  to  this  trend,  e.g.,  Finley,  et  al.,  1970.  These 
papers,  however,  do  provide  a  good  starting  point  for  the  selection  of  variables 
used  in  this  procedure. 

To  overcome  the  difficulty  noted  above,  this  paper  used  the  following  se¬ 
lection  strategy.  First  the  summary  literature  was  screened  for  suitable  train¬ 
ing  variables.  The  result  of  this  screening  was  the  division  of  variables  into 
three  basic  groups:  (1)  Variables  seeming  to  have  effects  on  training,  but 
with  little  empirical  support;  (2)  variables  having  effects  on  training  and 
having  empirical  support;  and  (3)  variables  have  some  empirical  support,  but 
with  very  mixed  results  with  regards  to  training.  Due  to  time  constraints,  it 
was  not  possible  to  conduct  independent  experiments  on  all  the  variables.  Thus, 
there  was  no  way  to  assess  their  consistent  impact  upon  the  training  processes. 
Consequently,  groups  1  and  3  were  excluded  from  further  consideration.  It 
should  be  noted,  however,  the  matrix  framework  that  is  used  does  permit  inclu¬ 
sion  of  new  variables  at  any  time.  In  addition,  the  authors  felt  that  until 
experience  is  gained  with  the  entire  procedure,  the  inclusion  of  too  many  vari¬ 
ables  could  be  counterproductive. 

Within  the  group  of  remaining  variables,  i.e.,  group  2,  there  were  still  a 
number  of  problems  to  be  resolved.  The  remaining  variables  were  divided  into 
smaller  groups  based  on  the  functional  role  they  played  in  the  experiments  from 
which  they  were  taken.  However,  before  this  could  be  done,  a  set  of  working 
definitions  had  to  be  developed  so  that  variables  could  be  compared  across 
studies.  Without  working  definitions,  the  variables  could  be  interpreted  in 
any  manner  and  could  leave  the  whole  procedure  too  ambiguous  to  use.  The  clas¬ 
sification  of  variables  was  accomplished  using  the  string  analogy  discussed  on 
pages  3-4.  Each  variable  was  evaluated  and  revised,  and  a  preliminary  set  of 
variables  was  developed. 

In  general,  the  variables  were  divided  into  four  classes,  depending  on 
their  domain  of  influence.  These  four  classes  correspond  to  the  program  and 
content  level  breakdowns  shown  on  page  3.  Variables  dealing  with  physical 
properties  of  real  equipment  were  classified  as  media  variables.  Variables 
dealing  with  interrelationships  between  media  items  or  social  structure  were 
placed  under  the  heading  of  method  variables.  Variables  dealing  with  the  type 
of  information  flowing  through  a  training  system  were  placed  under  the  heading 
of  content  variables.  Finally,  variables  dealing  with  the  interrelationships 
between  content  elements  were  placed  under  the  heading  of  structure  variables. 
The  following  examples  may  help  clarify  what  role  these  classes  of  variables 
play  in  the  training  process. 
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Example  1 

Media  variables  refer  to  actual  pieces  of  equipment  (or  even  humans)  that 
can  be  used  to  present,  accept,  or  evaluate  training  information.  These  are 
listed  in  Appendix  A.  (The  media  variables  were  extracted,  extensively  modi¬ 
fied,  and  combined  from  TAEG  Report  No.  16 7  and  Meister’s  Behavioral  Foundations 
of  System  Development . )  For  example,  a  television  set  is  a  fairly  sophisticated 
piece  of  equipment  having  the  capability  of  presenting  color,  sound,  and  move¬ 
ment  information.  A  workbook  is  also  a  physical  device,  but  with  less  capabil¬ 
ities.  In  another  sense,  even  an  instructor  can  be  viewed  as  a  training  medium 
with  information  handling  capabilities. 


Example  2 


Training  method  variables  (Appendix  B)  refer  to  the  patterns  and  sequence 
of  interactions  between  students  and  media.  For  example,  team  teaching  is  a 
pattern  of  interactions  between  more  than  one  instructor  and  more  than  one 
student.  The  variables  describing  this  pattern  of  interactions  are  method  var¬ 
iables.  The  generation  of  the  method  variables  required  developing  new  tech¬ 
niques  because  no  literature  was  found  characterizing  methods  in  a  form  com¬ 
patible  with  the  string  analogy.  Three  steps  were  required  initially.  First, 
the  five  questions  discussed  earlier  on  page  4  had  to  be  generated.  Sec¬ 
ond,  all  narrative  descriptions  of  methods  had  to  be  screened  and  rewritten  to 
reflect  the  key  differences  between  time.  (The  narrative  descriptions  of  meth¬ 
ods  were  extracted  and  modified  from  W.  R.  Tracey's  Designing  Training  and  De¬ 
velopment  Systems. 8)  Third,  all  methods  had  to  be  coded  in  terms  of  their  dif¬ 
ferences  based  on  the  five  questions.  (See  Appendix  B.) 

Another  technique  required  selecting  classes  of  method  variables  that  would 
link  the  task  environment  to  the  training  environment.  This  technique  consisted 
of  examining  the  sources  of  variation  in  operational  environments,  selecting 
variable  names  to  represent  that  variation,  examining  sources  of  variation  in 
the  method  descriptions,  selecting  variables  to  represent  that  variation,  and 
then  matching  the  two  sets  of  variables.  The  variables  that  overlapped,  i.e., 
linked  methods  and  operational  environments,  were  coded  in  matrix  form  and  ad¬ 
justed  until  the  space  of  methods  were  orthogonally  partitioned  by  variable 
values.  The  result  was  a  reduced  set  of  methods  which  could  be  shown  to  be 
operationally  different  from  one  another.  This  in  turn  reduced  the  total  set 
of  methods  and  refined  their  definitions  markedly.  Five  classes  of  method  var¬ 
iables  were  developed  and  they  included:  (1)  The  role  of  the  element,  (2)  the 
function  performed  in  the  role,  (3)  the  stability  of  the  function  over  time, 

(4)  the  effect  of  physical  environment,  and  (5)  the  effect  of  psychological 
environment.  (The  coding  of  these  variables  is  discussed  further  in 
Appendix  B . ) 


Example  3 


Content  variables  deal  with  the  particular  units  of  information  being 
taught  within  the  system.  These  include  information  such  as  task  type.  An 
example  of  a  content  variable  is  "mathematical  manipulation"  (Appendix  C) . 
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Example  4 

Structure  variables  refer  to  the  way  in  which  content  variables  are  related 
to  one  another.  They  are  concerned  with  such  things  as  repetition,  time  se¬ 
quencing,  similarity,  and  spacing  (Appendix  c) . 

To  generate  final  sets  of  all  variables,  another  problem  had  to  be  re¬ 
solved.  The  problem  was  the  way  in  which  the  variables  were  to  be  extracted 
from  task  analysis  descriptions,  since  any  predictive  technique  will  probably 
be  operating  with  input  information  in  a  variety  of  forms  in  early  states  of 
weapons  systems,  the  procedure  by  which  information  is  recoded  into  variables 
becomes  very  important. 

One  approach  is  to  recognize  that  all  textual  information  is  presented  via 
language,  and,  consequently,  the  information  carrying  elements  can  be  analyzed 
psycholinguistically .  In  linguistic  circles,  breakdown  of  text  seems  to  follow 
two  tacts.  The  first,  transformational  grammar,  generally  ascribed  to  Noam 
Chomsky,9  corresponds  roughly  to  structure  variables.  The  other  dominant  tact, 
case  structure  grammar,  emphasizes  the  meaning  (semantics)  of  the  information 
and  corresponds  more  directly  to  the  content  variables.  Typical  proponents  of 
this  view  are  Fillmore  and  Fodor.-*-0  In  general,  formal  linguistic  schemes,  al¬ 
though  having  considerable  value  for  future  applications,  such  as  real  time 
language  analyzers,  (Winograd,  1972) 11  do  not  have  immediate  applicability  for 
the  processing  of  task  information.  In  practice,  the  simple  procedure  of  break¬ 
ing  down  sentence  structure  into  noun,  verb,  and  object  with  adverbial  or  ad¬ 
jectival  modifiers  appears  quite  adequate  and  has  been  used  with  some  success 
in  various  classification  schemes.  Identification  of  critical  information  is 
often  quite  easy  for  a  trained  analyst  and  it  is  not  necessary  for  him  to  break 
task  information  into  linguistic  components.  However,  if  such  a  model  were  to 
be  entirely  automated  in  the  future,  a  linguistics  approach  might  be  of  con¬ 
siderable  value. 

Generally,  the  analysis  of  task  information  for  training  falls  into  the 
following  categories.  Task  verbs  are  useful  for  identifying  the  category  of 
action  taking  place  in  the  task.  When  paired  to  a  series  of  task  taxonomies, 
such  as  used  in  the  ARI  Training  Manager  Model, considerable  insight  can  be 
gained  into  the  types  of  skills  and  knowledges  required  for  a  group  of  tasks. 

By  pairing  the  task  types  to  operational  training  guidelines,  it  is  also  pos¬ 
sible  to  generate  unique  configurations  of  instructional  programs.  The  ARI 
Trainvice  Model1^  represents  one  study  in  this  direction.  Frequently,  nouns 
in  task  descriptions  apply  to  pieces  of  physical  equipment  or  units  of  infor¬ 
mation.  The  appropriate  category  is  clarified  by  adjectives.  Similar  clari¬ 
fications  for  verbs  occur  through  adverbs  and  various  other  modifying  phrases. 
The  wide  variety  of  descriptive  phrases  found  in  task  lists  seems  natural  be¬ 
cause  of  the  flexibility  of  the  English  language  structure.  A  given  object  or 
phrase  can  have  almost  any  combination  or  permutation  of  phrase  modifiers.  For 
example,  a  slide  projector  can  be  a  random  access,  high  speed,  black  plastic, 
carousel  projector,  or  it  can  be  a  low  speed,  sequential,  white  plastic,  indi¬ 
vidual  slide  projector.  In  this  example,  the  modifiers  set  bounds  upon  the 
equipment.  But  even  more,  in  the  case  of  a  task  analysis,  the  modifiers  spe¬ 
cify  conditions  for  either  a  functional  or  physical  fidelity  match.  The  most 
useful  insights  to  be  gained  from  linguistic  breakdown  concern  the  way  in 
which  the  variables  are  identified  to  allow  maximum  transfer  of  information 
from  the  task  descriptions  into  the  variable  description  lists. 


11 


After  the  initial  variable  screening  process  mentioned  earlier,  it  soon 
became  apparent  that  most  variables  fell  cleanly  into  categories  formulated  by 
the  string  analogy  and  could  be  identified  by  the  breakdown  of  the  task  infor¬ 
mation  in  the  simple  linguistic  model  by  verbs,  nouns,  and  modifiers. 

The  final  sets  of  variables  are  presented  in  Appendixes  A,  B,  and  C.  Sev¬ 
eral  things  should  be  noted  about  these  appendixes.  For  each  major  heading, 
such  as  MEDIA  RELATED  VARIABLES,  pages  41-43,  the  first  class  breakdown  of  vari¬ 
ables  is  in  terms  of  the  sensory  mode.  Depending  upon  the  training  function, 
the  trainee's  ability  to  perform  in  that  function  depends  on  two  things: 

(1)  The  sensory  modality  through  which  information  is  received,  and  (2)  the 
sensory  modality  by  which  he  expresses  his  response  to  that  information.  For 
example,  task  information  can  be  presented  to  him  through  visual,  auditory, 
tactile,  external  motion,  internal  motion,  olfactory,  or  taste  senses.  Simi¬ 
larly,  he  can  respond  auditorily,  physically,  or  internally.  Each  task  situa¬ 
tion  in  which  the  trainee  is  involved  must  flow  through  one  of  these  human  in¬ 
formation  processing  channels,  and,  consequently,  it  is  these  channels  that 
form  the  most  logical  basis  upon  which  to  break  down  the  sets  of  variables  for 
the  model.  At  the  start  of  each  set  of  variables  is  a  list  of  sensory  mediums 
by  which  the  training  information  can  be  exchanged.  A  given  task  or  training 
situation  may  use  one  or  any  combination  of  these  sensory  modalities.  A  list 
of  variables  exists  for  each  modality  which  corresponds  functionally  to  the 
previously  discussed  adjectival  or  adverbial  information  in  the  operational 
environments.  For  example,  six  types  of  "Visual  Form"  listed  under  MEDIA  RE¬ 
LATED  VARIABLES — STIMULI  CHARACTERISTICS  are  all  variables  for  visual  sensory 
information. 

Once  the  organizational  structure  was  chosen,  the  final  sorting  of  train¬ 
ing  variables  from  the  literature  became  straightforward.  The  string  analogy 
was  used  to  determine  the  type  of  process  each  variable  represented,  i.e.. 
Methods,  Media,  Content,  or  Structure.  The  linguistic  analysis  was  used  to 
determine  whether  it  was  directly  related  to  a  sensory  modality  or  whether  it 
was  a  modifier,  limiting  the  modality  and  its  range  of  operation.  The  variables 
were  refined  to  make  better  distinctions  between  nouns  (equipment)  and  modifiers 
(variables) .  The  result  of  this  refinement  was  a  list  of  training  equipment 
(Appendix  A- 36-40) ,  a  list  of  training  methods  (Appendix  B-54-76) ,  and  four  lists 
of  variables  broken  down  by  sensory  modalities  and  modifiers  (Appendixes  A  and 
B) .  These  lists  underwent  one  last  screening  to  assure  consistency  and  accu¬ 
racy  of  definitions. 


B.2.  IMPLEMENTING  THE  MODEL 

Once  the  variables  were  chosen,  it  was  possible  to  select  a  specific 
method  to  implement  the  model. 

The  model  begins  with  preparation  of  task  descriptions.  (Refer  to  Fig¬ 
ure  1.)  Task  information,  in  whatever  form,  is  broken  into  the  sets  of  vari¬ 
ables  by  the  following  procedure.  The  lists  of  variables  are  examined  to  de¬ 
termine  if  each  is  applicable  to  the  task  or  not.  Because  of  the  way  in  which 
the  variables  are  presented  (see  Appendix  A-41-43) ,  the  analyst  performing  the 
initial  coding  need  only  scan  the  lists  for  sensory  modalities  and  record  those 
that  apply.  If  a  sensory  modality  applies,  the  analyst  records  the  applicable 
modifiers  for  that  particular  modality.  If  a  sensory  modality  does  not  apply, 
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he  does  not  have  to  scan  the  modifiers  for  that  modality  because  they  are  elim¬ 
inated  from  consideration  automatically. 


The  next  step,  which  may  be  performed  either  serially  or  in  parallel  with 
variable  coding,  is  the  grouping  of  tasks  into  functional  units  for  blocked 
training.  (See  Figure  1.)  (There  are  many  ways  this  can  be  accomplished.  At 
present  no  one  procedure  has  been  chosen,  although  a  probable  course  could  be 
to  utilize  an  existing  program  developed  in  the  ARI  Training  Manager  Model . 

The  output  of  this  model  is  a  tree  diagram  of  related  tasks  based  on  similar 
personnel,  tasks,  and  equipment  functions.)  The  grouping  of  tasks  is  necessary 
to  assure  that  program  generation  processes  do  not  combine  tasks  in  totally  un¬ 
related  fashions.  The  grouping  should  be  based  on  logically  related  functional 
areas,  since  each  task  does  not  operate  in  a  vacuum,  but  rather,  occurs  within 
some  operational  context  or  environment.  Recall  that  it  is  the  operational  en¬ 
vironment  which  has  considerable  implications  for  the  instructional  method  to 
be  chosen. 

When  a  group  of  tasks  has  been  chosen  by  the  analyst,  the  method  and  the 
media  to  be  used  in  the  training  process  must  be  determined.  (See  Figure  1.) 

At  a  detailed  content  level  of  analysis,  the  particular  information  which  flows 
through  the  system,  as  well  as  the  structure  which  interrelates  that  informa¬ 
tion,  must  also  be  derived.  For  analysts  working  with  relatively  gross  task 
data,  this  level  of  analysis  is  not  practical.  The  manner  by  which  content  in¬ 
formation  should  be  structured  and  its  effect  on  performance  is  still  highly 
debatable.  Consequently,  although  the  variables  for  the  content  level  of  anal¬ 
ysis  were  developed  and  are  presented  in  Appendix  C,  they  are  not  used  in  the 
current  version  of  the  model.  When  specific  performance  predictions  do  become 
possible,  the  matrix  format  discussed  earlier  will  be  capable  of  easily  incor¬ 
porating  this  information.  Thus,  the  present  paper  focuses  its  attention  on 
those  areas  most  sensitive  to  cost,  namely  training  equipment  and  methods,  and 
their  subsequent  impacts  on  available  resources. 

After  tasks  are  grouped,  two  processes  begin.  For  each  task  grouping,  an 
appropriate  set  of  training  media  must  be  chosen.  (See  Figure  1.)  In  addi¬ 
tion,  operational  context  within  which  the  media  will  be  utilized  must  be  de¬ 
termined.  Just  as  the  trainee  interacts  with  actual  equipment  within  some 
operational  battlefield  scenario,  he  interacts  with  the  training  equipment  in 
the  scenario  of  an  instructional  method. 


B.2.a.  METHOD  SELECTION 


The  main  problem  to  be  overcome  in  method  selection  was  tne  identification 
of  suitable  variables  permitting  a  linkage  between  operationax  and  training 
context.  Direct  physical  fidelity  matching  used  in  the  construction  of  simu¬ 
lators  or  the  selection  of  training  equipment  was  not  feasible.  Although  the 
answer  seemed  to  lie  somewhere  in  the  functional  fidelity  area,  the  way  in 
which  this  linkage  could  be  accomplished  was  not  obvious.  Another  complicat¬ 
ing  factor  was  that  teaching  methods  were  not  clearly  defined  in  the  literature. 
The  problem  encountered  most  often  was  that  both  methods  and  equipment  were 
combined  and  defined  as  only  method.  Thus,  many  methods  were  actually  very 
similar,  but  seemed  different,  because  they  used  different  types  of  equipment 
to  perform  the  same  functions.  The  first  step,  therefore,  was  to  examine  the 
list  of  methods  discussed  in  the  literature  and  to  sort  them.  Methods  having 
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similar  patterns  of  interrelationships  were  grouped  together  under  the  same 
general  methodological  headings.  In  order  to  quantify  the  methods,  a  formal 
scheme  for  classifying  methods  was  created. 

The  string  analogy  proved  useful  for  accomplishing  this  task.  (See  the 
earlier  discussion  on  method,  page  10.)  Since  many  teaching  methods  center 
around  interpersonal  relationships  between  instructors  and  students,  it  was 
most  useful  to  characterize  the  variables  describing  these  relationships  in 
human  terms,  although  many  interactions  going  on  in  training  may  be  between 
man  and  machine.  Five  variable  areas  appeared  to  characterize  the  majority 
of  the  differences  among  methods.  These  areas  considered  five  characteristics 
of  the  training  program.  They  were: 

1.  Who  or  what  is  performing  the  action  or  processing? 

2.  How  many  times  does  this  action  or  processing  take  place  in  one 
instructional  method  sequence? 

3.  When  this  performance  takes  place,  in  what  order  is  the  person  or 
machine  in  the  sequence? 

4.  What  is  the  percentage  of  total  control  which  this  person  exercises 
over  the  instructional  process  taking  place  at  this  time? 

5.  What  percentage  of  the  total  instructional  time  is  taken  up  by  this 
activity? 

Defining  methods  in  terms  of  these  questions  led  to  the  schematic,  tabular 
definitions  found  in  Appendix  B-54-76.  In  all,  23  methods  were  analyzed  and 
given  both  narrative  and  quantifiable  definitions. 

Once  these  definitions  were  generated,  it  was  then  possible  to  establish  a 
mapping  between  the  operational  environment  and  the  training  environment  for  a 
given  group  of  tasks.  As  anticipated  earlier,  a  method’s  benefits  served  as  a 
logical  link  between  the  actual  operational  environment  and  training  method. 
Each  method  generally  has  ascribed  to  it  certain  idealized  characteristics. 
These  characteristics  are  the  reason  for  choosing  particular  configurations 
over  others.  (See  Appendix  B.)  Because  the  trainee  is  the  focus-,  one  logical 
way  to  link  the  functional  variables  and  training  methods  is: 

Step  One:  Identify  the  idealized  characteristics  which  expert  opinion 
attributes  to  each  method. 

Step  Two:  Based  on  the  characteristics  which  relate  the  trainee  to  his 
functional  context  in  the  classroom,  identify  common  variables  between  the 
functional  classroom  description  and  the  operational  context. 

Step  Three:  Retain  those  variables  common  to  both  situations  and  code 
them  in  the  process  matrix. 

The  result  of  this  process  can  be  seen  in  Appendix  B-79.  Along  the  Y-axis  are 
five  variables  that  were  found  to  overlap  both  situations.  They  are:  (1)  The 
role  of  the  performing  trainee  within  the  operational  context  of  the  system, 

(2)  the  particular  function  performed  by  the  trainee  within  that  role,  (3)  the 
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stability  of  that  function  over  time,  (4)  the  significance  of  the  physical  en¬ 
vironment  upon  performance  of  the  task,  and  (5)  the  significance  of  the  psycho¬ 
logical  environment  upon  performance  of  the  task.  Each  of  these  classes  of 
variables  could  theoretically  have  an  indefinite  number  of  levels.  The  number 
of  levels  chosen  for  this  paper  was  somewhat  arbitrary;  however,  the  number  se¬ 
lected  appeared  to  be  adequate  for  classifying  the  methods.  Role  was  divided 
into  three  levels  describing  functions  performed  by  the  trainee  in  his  actual 
duties  and  each  class  was  broken  down  as  follows:  (1)  supervisory,  where  one 
or  more  persons  are  involved;  (2)  team  performance,  where  the  trainee  has  equal 
roles  with  other  individuals  or  equipment;  and  (3)  individual  performance, 
where  the  trainee  performs  a  given  task  on  an  individual  basis.  The  function 
was  broken  into  four  levels  corresponding  to  the  type  of  operation  being  per¬ 
formed  in  the  role.  Many  more  levels  could  have  been  used,  for  example,  Willis 
and  Peterson's14  task  taxonomy  with  12  levels  could  have  been  chosen  equally  as 
well.  It  was  not  clear,  however,  how  the  advantages  attributed  to  specific 
methods  of  instruction  could  be  divided  much  finer,  and,  consequently,  only 
mental,  physical,  perceptual,  and  communicative  functions  were  considered  at 
present.  Stability,  which  reflects  how  often  the  function  performed  in  the 
operational  context  changed,  was  divided  into  two  levels,  high  and  low,  due 
to  lack  of  precise  causal  relations.  Physical  context,  referring  to  the  ef¬ 
fects  which  physical  characteristics  such  as  cramped  space  or  chemical  warfare 
environments  might  have  on  performance,  was  also  broken  into  two  levels.  Psy^ 
chological  context  includes  mental  influences,  such  as  peer  pressure,  which 
could  affect  job  performance  in  operational  situations.  Other  examples  are 
time  pressure  and  motivation.  Psychological  context  was  also  broken  into  two 
levels,  high  and  low.  Should  other  variables  prove  useful  in  the  future,  the 
matrix  structure  of  this  procedure  will  permit  their  rapid  inclusion.  The  en¬ 
tire  process  has  been  designed  to  accommodate  such  modifications  and  allow  for 
new  training  research  and  policy  changes. 

Each  of  the  23  methods  mentioned  earlier  was  coded  in  terms  of  its  ability 
to  manipulate  each  of  the  five  variables  for  a  trainee.  The  matrix  is  found  in 
Appendix  B-79,  with  each  column  corresponding  to  one  of  the  methods  and  each 
row  to  one  of  the  levels  of  the  functional  context  variables.  If  a  given  method 
includes  more  than  one  level  of  a  variable,  it  is  coded  in  each  row  where  it 
applies.  As  can  be  seen  from  Appendix  B-79,  each  method  differs  somewhat  in 
terms  of  which  functional  context  variable  it  manipulates.  Thus,  it  is  possi¬ 
ble  to  match  the  fielded  operational  context  within  which  the  trainee  will  work 
with  the  variables  describing  the  method  which  can  produce  similar  training  en¬ 
vironments.  It  is  recognized  that  for  various  reasons  a  particular  school  may 
not  desire  to  train  an  individual  in  a  manner  that  corresponds  directly  to  his 
or  her  field  environment.  In  such  a  case,  adjustments  can  be  made  to  the  ma¬ 
trix  to  represent  differences  by  changing  their  relative  weights  as  shown  ear¬ 
lier.  One  advantage  of  the  current  approach  is  that  training  is  oriented  toward 
battlefield  training,  and  any  change  would  require  explicit  changes  in  matrix 
weights,  thus  highlighting  cases  where  battlefield  performance  was  not  the  final 
goal . 


As  more  is  learned  about  the  relative  advantages  or  disadvantages  of  each 
training  method  from  actual  performance  data,  the  matrix  can  be  selectively 
weighted  to  produce  a  better  and  better  fit. 
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B.2.b.  MEDIA  SELECTION 


The  selection  of  training  media  follows  much  the  same  logic  as  method  se¬ 
lection,  except  that  the  media  selection  process  involves  three  subprocesses 
corresponding  to  different  information  flows  taking  place  in  a  training  pro¬ 
gram.  These  flows  are  stimulus  presentation,  response  acceptance,  and  feedback 
evaluation.  Each  complete  instructional  sequence  includes  one  or  all  of  these 
components.  The  three  subprocesses  may  be  performed  by  a  single  piece  of  equip¬ 
ment,  a  single  individual,  or  a  combination  of  both.  For  example,  stimulus  in¬ 
formation  may  be  presented  by  a  radar  screen,  the  response  of  the  operator  to 
that  stimulus  may  be  an  audio  reply  through  a  radio  net,  and  the  evaluation  of 
the  correctness  of  the  response  may  be  a  verbal  one  made  by  a  commanding  offi¬ 
cer  standing  nearby.  On  the  other  hand,  the  entire  sequence  may  take  place  en¬ 
tirely  between  the  operator  and  a  computer  controlled  threat  evaluation  system 
contained  within  the  weapon  itself. 

This  kind  of  flexibility  points  out  a  problem  with  matrix  or  network  se¬ 
lection  systems  which  have  only  a  single  media  selection  process.  Because  one 
matrix  then  includes  all  the  selection  subprocesses ,  it  is  inherently  biased  in 
favor  of  a  single  device  having  the  largest  number  of  capabilities,  simply  be¬ 
cause  that  device  is  most  flexible.  The  result  is  that  real  equipment  or  simu¬ 
lators  are  almost  always  chosen,  even  though  several  smaller,  but  less  expen¬ 
sive  components  can  be  fitted  to  duplicate  information  flows  performed  in  the 
same  task.  In  this  paper  each  of  the  three  subprocesses  has  its  own  set  of 
relevant  variables.  The  complete  set  of  variables  for  each  subprocess  is  found 
in  Appendix  A.  Even  though  each  subprocess  has  its  own  special  characteristics, 
there  is  variable  overlap.  The  set  of  variables  for  media  corresponds  directly 
to  the  sensory  characteristics  of  the  training  situations,  i.e.,  stimulus,  re¬ 
sponse,  or  feedback.  The  process  matches  the  elements  having  physical  and 
functional  fidelity  in  the  operational  equipment  with  the  elements  of  the  train¬ 
ing  environment  in  contrast  to  method  selection,  where  the  determination  of  a 
method  was  more  directly  related  to  social  interactions  between  students  and 
the  persons  or  equipment  performing  the  media  functions. 

Media  selection  is  made  by  comparing  coded  task  data  with  selection  ma¬ 
trices  in  the  same  manner  as  the  earlier  matrix  example.  These  selection  ma¬ 
trices  are  shown  in  Appendix  A-44,  45,  48,  and  51.  At  this  time,  there  is  not 
enough  data  to  put  specific  weighting  values  in  the  matrix,  so  1  and  0  are  used 
to  represent  applicability  or  nonapplicability,  respectively. 


B.2.C.  CONTENT  AND  STRUCTURE  SELECTION 


The  two  sets  of  processes  which  have  not  been  considered  up  to  this  point, 
involve  those  variables  relating  to  the  information  flowing  through  the  train¬ 
ing  equipment  and  their  configuration.  These  were  called  content  and  structure 
variables.  The  entire  area  dealing  with  individual  differences  and  their  ef¬ 
fects  and  interactions  with  course  material  is  hotly  contested.  For  a  predic¬ 
tive  methodology,  it  would  appear  to  be  premature  to  attempt  anything  more  than 
general  statements  in  this  area.  In  the  present  model,  several  procedures  were 
considered  to  handle  these  remaining  areas.  The  procedure  that  seemed  most  ap¬ 
plicable  for  future  use  consisted  of  using  a  verb  dictionary  to  select  informa¬ 
tion  content  presented  in  each  task  set.  Each  entry  in  the  verb  dictionary  is 
paired  to  a  categorical  heading  or  task  type.  Associated  with  each  task  type 
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mi  is. 


were  lists  of  guidelines  or  flowcharts  which  served  as  a  coursewriter *s  aid  for 
generating  courseware  material.  Currently,  ARI  has  an  ongoing  research  effort. 
Training  Managers  Model,  handling  the  problem  in  this  manner.  Another  effort 
aimed  at  the  prediction  of  training  device  effectiveness,  the  Trainvice  Model, 
also  contains  guidelines  which  could  be  paired  to  the  same  categories.  It  was 
felt  that  these  efforts  are  not  sufficiently  developed  to  be  useful  for  early 
training  prediction  although  they  have  important  applications  at  other  levels 
of  training  development.  Consequently,  this  paper  strives  only  to  predict  the 
grosser  elements  of  the  training  program  related  to  costing  media  and  method 
choices.  Because  of  this  limitation,  predicting  overall  field  effectiveness  is 
limited . 


B.2.d.  EFFECTIVENESS  PREDICTION 


Effectiveness  prediction  is  probably  the  most  difficult  problem  to  solve 
in  CTEA.  An  important  contributor  to  this  difficulty  is  ambiguity  about  the 
range  of  effects  which  should  be  considered.  Specifically,  how  extensively 
should  effectiveness  be  defined  for  CTEA  in  contrast  with  COEA  or  battlefield 
simulation?  Although  there  is  no  doubt  that  training  directly  influences  field 
performance,  the  linkage  between  training  programs  and  terminal  battlefield 
performance  is  much  more  tenuous.  For  example,  studying  10  hours  using  a  car¬ 
ousel  projector  instead  of  12  hours  using  e  textbook  may  have  a  ]ater  impact  on 
battlefield  performance,  however,  the  chain  of  causal  events  is  contaminated  by 
so  many  other  factors  that  it  seems  futile  to  attempt  predictive  links  at  such 
a  level.  What  can  be  done  is  to  break  the  causal  chain  into  a  series  of  larger, 
more  controllable,  sequences  of  events,  and  attempt  to  predict  them.  Later, 
that  information  may  be  used  in  combination  with  other  factors  as  the  basis  for 
predicting  overall  battlefield  performance.  The  implication  of  this  for  CTEA 
is  that  a  measure  of  effectiveness  must  at  present  be  taken  from  the  battle¬ 
field  and  focused  on  the  school  where  the  degree  of  control  is  the  greatest. 

The  measure  chosen  in  this  paper  is  even  more  restricted.  What  can  be  con¬ 
trolled  is  the  efficiency  with  which  a  projected  training  program  is  developed. 
Theoretically,  there  are  least  five  sources  of  information  in  this  model  from 
which  an  effectiveness  metric  can  be  fashioned.  All  four  process  selection  ma¬ 
trices  and  the  variables  themselves  could  be  used  as  predictors  using  a  form  of 
regression  equation.  Many  other  pieces  of  information,  also  possible,  are  not 
presently  being  considered,  such  as  managerial  constraints,  personnel  charac¬ 
teristics,  teacher  characteristics  and  a  host  of  aptitude  by  treatment  inter¬ 
action  variables.  “  A  practical  choice  for  the  present  is  to  record  alternative 
costs  produced  in  the  generation  of  a  projected  program.  Individual  decision 
costs  are  then  combined  to  determine  the  overall  loss  for  a  predicted,  but  con¬ 
strained,  training  program  versus  an  "ideal"  program  with  no  constraints.  Al¬ 
though  such  a  metric  supplies  no  information  about  the  absolute  scalar  proper¬ 
ties  of  a  number  so  generated,  it  dc^.s  provide  a  relative  standard  of  comparison 
for  several  training  programs  generated  using  the  same  method.  This  has  poten¬ 
tial  advantages  not  only  for  the  comparison  of  alternative  programs  within  a 
given  weapon  system,  but  also  between  systems. 

In  this  paper  the  metric  is  referred  to  as  an  efficiency  ratio.  It  repre¬ 
sents  a  value  which  is  composed  of  the  total  matrix  "fit"  of  a  projected  train¬ 
ing  program  with  real  world  constraints,  divided  by  the  fit  of  an  idealized 
program  which  is  subject  to  no  constraints.  The  value  is  generated  in  the  fol¬ 
lowing  manner.  First,  the  selection  matrix  scores  for  each  media  matrix  are 
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recorded-  (See  vector  c  in  the  example  in  Figure  2.)  Repeated  over  all  tasks 
this  produces  three  new  matrices  of  c  vectors  where  the  rows  now  correspond  to 
tasks  and  the  columns  correspond  to  the  degree  of  fit  which  that  task  had  with 
each  of  the  potential  training  media.  If  the  largest  sum  in  each  of  these  rows 
is  chosen,  a  large  number  of  different  equipment  choices  would  be  produced 
spread  over  all  the  tasks  which  are  being  considered.  Thus,  for  a  sample  group 
of  six  tasks  there  might  be  six  different  media  choices  for  each  of  the  three 
media  functions  or  a  total  of  18  different  devices.  If  cost  were  no  object, 
and  there  were  no  resource  or  time  constraints,  the  ideal  choice  of  media  would 
be  all  18  devices.  The  total  of  the  equipment  fit  scores  for  those  18  pieces 
then  represents  an  ideal  choice  without  constraints.  This,  of  course,  is  not 
realistic,  since  there  are  always  constraints  that  enter  into  any  decision.  As 
constraints  are  applied,  the  viability  of  a  particular  piece  of  equipment  for  a 
particular  task  may  change,  just  as  it  did  in  Figure  2.  For  example,  even 
though  it  may  not  be  optimal  equipment,  the  training  manager  may  have  on  hand  a 
particular  training  device  and  want  to  include  that  equipment  in  the  program, 
rather  than  buy  a  newer,  more  expensive  one.  Perhaps  the  piece  of  equipment 
permits  combining  several  tasks  rather  than  individually  training  each.  In  the 
above  discussion,  this  corresponds  to  choosing  one  of  the  other  sums  in  a  col¬ 
umn  representing  the  particular  piece  of  equipment  the  manager  desires  to  use. 

If  he  were  now  to  do  this  for  all  six  tasks,  i.e.,  force  the  choice  of  the  same 
piece  of  equipment,  the  sum  of  the  numbers  across  tasks  would  be  much  lower. 
Thus,  the  efficiency  of  that  particular  decision  would  be  lower  than  the  ideal 
case,  as  would  all  other  potential  combinations.  If  this  value  is  then  divided 
by  the  ideal  first  value,  what  would  be  produced  is  a  percent  ratio  of  actual 
to  ideal,  or  in  the  terminology  used  above,  an  efficiency  ratio.  This  is  the 
approach  used  in  this  model.  (See  Figure  5.) 

In  actuality  this  procedure  becomes  much  more  complicated,  when  many  tasks, 
media  functions,  and  media  types  are  considered.  The  actual  program  is  some¬ 
what  analogous  to  the  statistical  technique  of  hierarchical  grouping  analysis, 
where  the  ideal  program  is  a  starting  point,  and  the  number  of  potential  media 
is  gradually  reduced  in  a  step-wise  function  while  keeping  track  of  the  tasks 
which  are  subsummed  under  each  function  and  media  type.  As  the  number  of  redia 
is  reduced,  a  plot  is  made  of  the  efficiency  ratios  for  each  combination.  The 
result  illustrated  in  Figure  5  is  a  graphic  aid  to  allow  an  analyst  to  identify 
the  point  at  which  further  trade  offs  in  equipment  begin  to  have  significant 
effects  on  the  training  fidelity  of  the  system.  This  point  is  represented  by  a 
sudden  drop  in  the  ratio  value.  This  will  become  more  obvious  in  the  example 
given  later. 


B . 2 .e .  THE  COSTING  PROCEDURE 

A  final  problem  consists  of  the  costing  procedure  by  which  the  output  of 
the  method  and  media  estimations  are  analyzed.  This  model  uses  a  costing  pro¬ 
gram  originally  developed  by  the  Navy  for  this  purpose.7  Equipment  and  method 
variables  are  costed  in  terms  of  a  set  of  37  variables  specifically  related  to 
a  cost  accounting  procedure.  These  components  are  then  put  into  a  computer 
program  which  calculates  the  costed  output.  This  cost  model  is  currently  in 
use  by  the  U.S.  Army  Air  Defense  School  for  overall  training  estimates  in  the 
ROLAND  system  and  is  being  considered  for  Army-wide  application.  A  modified 
version  of  this  program  is  proving  helpful  in  aiding  decision  makers  after  a 
training  program  has  been  developed.  An  example  of  a  preliminary  application 
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Step  1  -  Group  all  (c)  vectors  (see  Figure  4 )  into  3  matrices  corresponding 

to  stimulus,  response,  and  feedback  functions  for  media  selection. 


Task  1 
Task  2 


Stimulus 


C11  C12‘  rCln 


Response 


C11  C12'  rCln 


Feedback 
C11  C12'[ ‘Cln 


Task  n 


C  .  C  .C 
nl  n2  nn 


Cnl  Cn2r  'Cnn 


r  r  C 
nl  n2‘--  nn 


Step  2  -  For  each  task  choose  the  largest  value  of  C..,  i.e.,  L...  and  add 

I  J  '  J 

over  all  tasks.  This  produces  3  sums,  one  for  each  matrix:  Sstimulus 
Response 1  anc*  5 feedback' 

^stimulus  -  LIask  1  +  Hask  2  +''‘^Task  n 

Response  =  LJask  1  +  LTa$k  2  +--*4ask  n 

feedback  =  LTask  1  +  LTask  2  +'"LTask  n 


Step  3  -  Add  the  three  S  values  together  for  a  measure  of  the  overall  fit 
of  the  C.  •  chosen  for  all  tasks  and  all  functions. 

*  J 

Overall  Measure  =  S„.  .  llllic  +  „  +  S-  ...  . 

stimulus  response  feedback 

Step  4  -  Record  the  overall  measure.  Repeat  Steps  1-3,  after  removing 
the  media  in  each  stimulus,  response,  and  feedback  matrix  (Step  1)  having 
the  lowest  column  sum  across  all  tasks.  Record  this  new  overall  measure, 
and  repeat  until  all  media  have  been  considered. 

Figure  5.  The  generation  of  a  single  efficiency  ratio 
and  plot  of  possible  ratios. 
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step  5  _  plot  the  ratios  of  the  overall  measure  values,  using  the  first 
value  as  the  denominator  for  all  the  remaining  values. 


First  Overall  Measure 
First  Overall  Measure 


Second  Overall  Measure 
First  Overall  Measure 


Some  Value  <  1 


Ratio  n 


Last  Possible  Overall  Measure 
First  Overall  Measure 


Some  Value  «  1 


Ratio 

Value 


1.0 


.5 


\ 


\ 


*  Value  Representing  a  Single  Efficiency 
\Ratio 


0 


2 


3 


4 . n 


Ratio  Number 


Figure  5  (Continued) 


of  the  Navy  program  is  available  as  part  of  an  ARI  Technical  Memorandum*6  writ¬ 
ten  in  late  1975  as  part  of  a  preliminary  feasibility  study  of  matrix  selection 
concept  and  costing  logic  in  COEA. 


C.  AN  EXAMPLE 

In  order  to  pull  together  the  prediction  techniques,  a  simplified  example 
is  given.  Figure  6  should  be  continually  referenced  for  a  clearer  picture  of 
the  interactions  among  components.  To  make  the  example  as  simple  as  possible 
only  a  single  task  is  considered  and  traced  through  the  model.  This  will,  how¬ 
ever,  artificially  constrain  certain  areas.  These  will  be  elaborated  as  they 
are  reached.  The  following  is  a  sample  task: 

Assume  the  trainee  is  to  be  used  in  an  Air  Defense  System  consisting 
of  an  interactive  radar  scope  connected  to  a  computer  and  a  print  out 
device  for  typed  material.  He  is  to  detect  malfunctions  in  the  system 
resulting  in  unpredictable  error  messages  appearing  on  the  screen. 

Upon  detection  of  an  error  he  is  to  use  the  keyboard  printer  to  make 
another  display  scope  operational  and  is  to  verify  the  result  of  the 
change  on  the  keyboard  printer  output  produced  by  the  computer. 

The  task  description  is  first  compared  to  the  variables  listed  under  MEDIA 
RELATED  VARIABLES — STIMULUS  CHARACTERISTICS,  Appendix  A-41-43.  Note  that  compar 
ison  of  the  task  and  variable  list  numbers  1-7,  shows  only  variable  1  (visual 
symbolic)  to  be  applicable.  Because  sensory  modalities  numbers  2-7  do  not  ap¬ 
ply,  they  can  be  eliminated  along  with  variables  25-30  which  apply  to  audio 
stimuli  only.  Variables  8-24  and  31-39  must  now  be  examined  for  their  appli¬ 
cability  to  the  task  description.  If  a  variable  is  shown  to  be  applicable  it 
is  scored  with  a  1,  and  if  it  is  shown  to  be  non-applicable,  it  is  scored  with 
a  0.  These  evaluations  are  kept  on  separate  task  coding  sheets,  like  the  one 
found  on  page  39. 

The  example  involves  observation  of  a  visual  display.  Normally,  the  "Vis¬ 
ual  Forms,"  variables  8-13,  which  can  be  presented  on  displays,  include  alpha¬ 
numeric,  symbolic,  and  graphic  information  (8,  9,  and  10) ,  but  do  not  include 
other  forms,  such  as  pictorial,  solid  object,  or  environmental  (11,  12,  and 
13) .  Variables  8,  9,  and  10  would  each  be  coded  as  1  on  the  coding  sheet, 
while  variables  11,  12,  and  13  would  be  coded  as  0.  A  similar  procedure  is 
used  for  all  other  relevant  variables.  For  instance,  a  message  found  on  a  dis¬ 
play  may  change  at  any  time,  and  is  therefore  not  static  (31).  Messages  pre¬ 
sented  on  the  displays  are  also  quite  random  in  time  (33) ,  rather  than  ordered 
(32) .  Therefore,  variables  31  and  32  would  be  scored  with  a  0  and  33  would  be 
scored  with  a  1.  Once  the  entire  list  of  applicable  variables  has  been  scored 
on  the  coding  sheet  in  this  manner,  the  task  will  have  been  described  for  the 
stimulus  situation  which  initiates  the  operator  actions.  (See  the  first  row  of 
the  Sample  Coding  Sheet  for  the  applicable  variables  for  this  sample  task  on 
page  39.  This  entire  row  of  variables  corresponds  to  the  row  vector  for  the 
stimulus  situation.) 

The  next  step  is  to  generate  the  row  vector  corresponding  to  the  MEDIA 
RELATED  VARIABLES — RESPONSE  CHARACTERISTICS,  Appendix  A-46-47.  Only  the  physi¬ 
cal  properties  of  the  equipment  which  receive  the  operator's  response  to  the 
stimulus  are  considered.  In  this  example,  note  that  the  operator's  response  is 
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overt,  i.e.,  visible,  and  involves  the  manipulation  of  buttons  on  a  keyboard 
printer.  Thus,  variables  1,  2,  4,  and  5  are  coded  as  a  0,  and  3  is  coded  as  a 
1  on  the  coding  sheet.  Likewise,  the  responses  are  not  found  to  require  major  r 

physical  movements  and  are  coded  as  weak  intensity  responses  (6)  .  since  the  i 

trainee  is  manipulating  equipment,  the  responses  he  produces  will  be  dynamic-  j 

ordered  (9)  and  the  equipment  must  be  capable  of  accepting  such  a  response. 

Similarly,  the  response  rate  would  probably  be  fast  (12) ,  the  number  of  response 
channels,  limited  (13) ,  and  the  response  distribution,  individual  (15) .  (See 
the  second  row  of  the  Sample  Coding  Sheet  for  the  applicable  variables  for 
this  sample  task.  This  entire  row  of  variables  corresponds  to  the  row  vector 
for  the  response  situation.) 

The  task  is  now  coded  in  the  same  manner  for  feedback  variables,  Appen¬ 
dix  A-49-50.  In  this  example,  the  medium  of  information  is  written  form  (3) . 

The  source  of  feedback  is  intrinsic  (10) ,  the  type  of  feedback  is  both  response 
consequences  (14)  and  system  status  (15),  and  the  feedback  distribution  is 
again  individual  (16) .  (See  the  third  row  of  the  Sample  Coding  Sheet,  page  39, 
for  the  applicable  variables  for  this  example.  This  entire  row  of  variables 
corresponds  to  the  row  vector  for  the  feedback  situation.) 

After  the  task  information  has  been  recoded  into  variables  directly  re¬ 
lated  to  media  choice,  another  evaluation  is  required.  A  group  of  tasks  would 
now  be  evaluated  in  terms  of  operational  context  variables  for  method  selection. 

Because  this  example  only  deals  with  a  single  task,  this  coding  is  artificial, 
since  normally  a  method  would  not  be  chosen  for  a  single  task  situation,  but 
rather  for  the  overall  environment  within  which  a  cluster  of  related  tasks 
would  be  performed.  The  method  variables  are  found  in  Appendix  B-77.  (Note; 

In  Appendix  C-82-86  there  is  another  set  of  variables  broken  down  by  stimulus, 
response,  and  feedback  functions.  These  variables  were  drawn  from  the  summary 
literature,  but  are  not  used  in  the  method  selection  matrix.  A  mapping  did  not 
seem  possible  between  the  operational  context  and  these  variables  at  this  time. 

However,  they  are  included  in  Appendix  C  as  a  resource,  having  undergone  the 
same  sorting  and  reduction  processes  as  the  other  variables.  These  variables 
might  be  useful  in  the  future  generation  of  a  predictive  regression  equation 
for  training  methods.) 

Coding  the  method  process  requires  the  same  mechanical  procedure  used  to 
code  the  media  subprocesses.  For  this  sample  task,  the  role  performed  by  the 
operator  in  the  real  environment  is  individual  performance  (3) .  The  functions 
performed  in  the  role  are  primarily  mental,  in  terms  of  analyzing  printouts 
(4);  physical,  in  terms  of  selecting  display  console  options  (5);  and  percep¬ 
tual,  in  observing  the  displays  (6).  The  function  is  basically  stable  (9), 
since  it  does  not  change  often.  The  impact  of  the  physical  context  (10)  and 
psychological  context  (12)  is  low.  (See  the  fourth  row  of  the  Sample  Coding 
Sheet  for  the  applicable  variables  for  this  sample  task.  This  entire  row  of 
variables  corresponds  to  the  row  vector  for  the  functional  context  variables 
for  method.) 

Because  the  model  does  not  presently  use  content  or  structure  information 
no  other  codings  are  necessary  for  this  sample  task.  (Content  and  structure 
variables  are  listed  in  Appendix  C-87-98  only  as  a  future  research  resource.)  ; 
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Selection  processes  can  now  be  performed.  First,  the  media-stimulus  se¬ 
lection  matrix,  shown  in  Sample  Figure  7,  is  premultiplied  by  the  stimulus  row 
vector.  The  result  of  this  multiplication  is  the  vector  of  sums  (a) ,  which  re¬ 
flects  how  well  each  device  fits  the  conditions  described  by  the  task  vector. 
Vector  (a)  can  be  found  on  the  bottom  row  of  the  sample  figure.  The  pieces  of 
equipment  showing  the  "best  fit"  are  carrel  (19) ,  dial  access  information  re¬ 
trieval  system — audio/visual  (20) ,  and  computer  (32) .  Because  only  a  single 
task  has  been  used  in  this  example,  more  than  one  piece  of  equipment  shows  the 
same  value.  However,  if  more  tasks  had  been  used,  the  selection  process  would 
have  narrowed  down  the  number  of  "best  fit"  pieces  of  equipment.  If  there  are 
a  number  of  pieces  of  equipment  that  have  the  same  value,  the  equipment  that 
costs  the  least  would  be  the  one  selected.  Complete  descriptions  of  the  three 
devices  selected  are  found  in  Appendix  A-36-40. 

In  a  similar  fashion,  the  response  matrix,  found  in  Sample  Figure  8,  is 
premultiplied  by  the  response  row  vector.  The  result  of  this  multiplication  is 
the  vector  of  sums  (b) .  Vector  (b)  can  be  found  on  the  bottom  row  of  the  sample 
figure.  The  largest  value  in  the  vector  is  6,  which  corresponds  to  carrels 
(19) ,  demonstrators  (33) ,  mock-up  and  panel  (34) ,  and  operational  equipment 
(35) .  The  description  for  each  of  these  devices  can  be  found  in  Appendix  A-36-40. 

Finally,  the  feedback  vector  is  multiplied.  Its  vector  of  sums  (c)  can  be 
found  on  the  bottom  row  of  Sample  Figure  9.  The  "best  fit"  media  are  computer 
(32)  and  operational  equipment  (35) . 

Now  the  method  can  be  selected.  Using  the  operational  context  (method) 
row  vector,  a  vector  of  sums  (d)  is  produced  from  premultiplication  of  the 
method  matrix.  (See  Sample  Figure  10.)  The  method  selected  is  tutoring  (13) . 

Its  description  is  given  in  Appendix  B-66,  both  in  the  detailed  string  analogy 
and  narrative  formats. 

For  this  example,  the  complete  package  would  consist  of  a  carrel  for  a 
stimulus  presentation  device,  the  carrel  for  a  response  acceptance  device,  and 
a  computer  for  a  feedback  evaluation  device.  The  "best  fit"  method  would  be 
tutoring.  If  "instructor"  had  been  included  as  a  type  of  media  in  the  selec¬ 
tion  matrices  for  media,  the  probability  that  an  instructor  would  have  been 
chosen  over  a  computer  for  feedback  evaluation  is  high,  since  he  probably  costs 
less  and  is  much  more  flexible.  In  addition,  an  instructor  would  fit  well  with 
the  tutorial  method  selected. 

This  example  has  been  made  artificially  simple  and,  therefore,  is  limited. 
Only  one  task  at  a  time  is  being  considered  in  the  selections  of  stimulus,  re¬ 
sponse,  and  feedback  equipment  and  method.  Because  only  a  single  task  is  used, 
graphic  analytic  aids  for  the  manager  cannot  be  shown.  If  several  tasks  had 
been  used,  there  would  have  been  three  vectors  of  sums  for  each  media  subproc¬ 
ess.  Each  one  would  have  been  grouped  in  a  matrix  and  have  been  subjected  to 
the  procedure  that  generates  the  efficiency  ratios.  When  a  large  number  of  un¬ 
related  tasks  is  considered,  a  number  of  equipment  choices  may  be  generated, 
but  when  a  small  number  of  highly  related  tasks  is  considered,  the  result  is 
often  a  single  piece  of  equipment. 
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Figure  7.  Media  selection  matrix. 
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Figure  7  (Continued) 
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Figure  9.  Media  selection  matrix. 


FIGURE  (10) 
SELECTION  MATRIX 


S WM 


Cost  information  would  also  be  required  for  the  example.  First,  a  vari¬ 
able  sheet,  like  that  shown  in  Appendix  D,  would  be  filled  out  for  each  piece 
of  equipment.  This  procedure  is  included  because  of  its  implications  for  in¬ 
cluding  the  remaining  variables,  such  as  number  of  instructors,  floor  space, 
student  turnover,  etc.  This  data  is  punched  out  and  submitted  for  computer 
analysis.  The  output  of  this  analysis,  combined  with  the  efficiency  ratio, 
constitutes  part  of  the  input  for  a  CTEA  of  a  given  weapon  system. 

In  its  current  state,  the  model  is  now  undergoing  its  first  stages  of 
testing.  The  costing  portion,  as  mentioned  earlier,  is  being  actively  used  by 
the  Army  Air  Defense  School  at  this  time.  The  selection  portion  has  been  coded 
into  a  computer  program  suitable  for  running  on  an  IBM  360-65  operating  system. 
It  is  currently  being  used  by  the  TRADOC  System  Analysis  Activity  (TRASANA) 
Training  Effectiveness  Analysis  branch  for  initial  estimation  in  a  variety  of 
CTEA  studies.  Greater  model  specification  should  result  as  experience  is 
gained.  It  is  anticipated  that  weighting  vectors  for  the  processes  resulting 
from  managerial  variables,  gross  cost  factors,  and  personnel  characteristics 
will  be  included  and  continually  refined. 

Many  problems  still  remain  in  the  training  program  formulation  area.  It 
is  hoped  that  the  present  approach  may  provide  a  useful  starting  point  for  the 
CTEA  requirements  dealing  with  the  early  prediction  of  training  programs  and 
help  with  the  many  trade-off  decisions  that  must  be  made  prior  to  full  scale 
development. 
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APPENDIX  A 


MEDIA 


1.  Descriptions  of  Training  Equipment  and  Materials  (A-2-6) 

2.  Media  Related  Variables 

a.  Stimulus  Characteristics  (A-7-9) 

b.  Media  Selection  Matrix  -  Stimuli  Variables  (A-10-11) 

c.  Response  Characteristics  (A-12-13) 

d.  Media  Selection  Matrix  -  Response  Variables  (A- 14) 

e.  Information  Feedback  Logic  (A-15-16) 

f.  Media  Selection  Matrix  -  Feedback  Variables  (A-17) 
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TE3ET? 


TRAINING  EQUIPMENT  &  MATERIALS 


Visual  Only  Systems 

1.  Case  Study  Folders  -  Folders  with  illustrative  studies  showing  cause 
and  effect  information,  may  include  pictures,  graphs,  maps,  charts,  etc. 

2.  Flash  Cards  -  A  set  or  deck  of  cards  designed  to  present  information 
to  a  group  or  individual  student. 

3.  Printed  Materials  -  Handouts  -  Handouts  are  a  class  of  printed 
materials  issued  to  a  student  for  his  use  and  retention  to  augment  regular 
instructional  materials.  They  are  usually  instructor  prepared,  machine 
copied  materials  of  one  or  two  pages  highlighting  specific  topics  or 
updating  existing  materials. 

4.  Printed  Materials  -  Performance  Aids  -  Performance  aids  are  a  class 
of  printed  materials  that  display  data  to  aid  in  job  performance  or  to 
identify  facts  or  background  information.  They  include  conversion 
tables,  data  charts,  schematic  diagrams,  equipment  test  tolerance  matrices, 
checklist  routines,  maps,  and  the  like. 

5.  Printed  Materials  -  Reference  Books  -  Reference  books  are  a  class  of 
printed  materials  used  to  identify  certain  facts  or  for  background 
information  such  as  dictionaries,  encyclopedias,  technical  publications., 
guides  or  manuals. 

6.  Printed  Materials  -  Workbooks  and  Exercises  -  A  class  of  printed 
materials  used  to  augment  or  replace  instructional  texts  by  providing  a 
mix  of  text  information  and/or  practice  exercises  and  quizzes. 

7.  Printed  Materials  -  Textbooks  -  Textbooks  are  a  class  of  printed 
material  dealing  with  a  subject  of  study  and  used  as  a  principal  source 
of  organized  information. 

8.  Programmed  Text  -  A  printed  text  containing  frames  of  information, 
either  in  the  form  of  questions  requiring  the  trainee  to  construct  simple 
written  responses,  multiple  choice  questions,  etc.  The  material  is 
carefully  sequenced,  tested,  and  revised  to  ensure  that  a  specific  student 
population  will  achieve  stated  behavioral  objectives  with  a  predetermined 
level  of  success. 

9.  Filmstrip  Projection  System  -  A  single  frame  projector  or  attachment 
thereto  that  will  accept  a  filmstrip  format  and  project  the  film 
images  upon  a  viewing  screen. 

10.  Microform  -  Microimagery,  such  as  microfilm,  used  as  a  medium  of 
instruction. 
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11.  Overhead  Projection  System  -  A  system  consisting  of  a  horizontal 
stage  projector  designed  to  use  a  vertical  throw  for  focusing  an 
enlarged  transparency  image  upon  a  projection  screen. 

12.  Slide  Projector  System  -  2"  x  2"  -  A  class  of  single  frame  picture 
projectors  that  will  accept  a  standard  2"  x  2"  slide  and  project  the 
contained  image  upon  a  viewing  screen. 

Audio  Only  Systems 

13.  Audio  Disc  System  -  An  audio  system  that  uses  a  record  player  and 
sound  recorded  on  a  disc  (record). 

14.  Audio  Tape  System  -  An  audio  system  that  uses  a  tape  recorder/ 
reproducer  to  record  sound  on  magnetic  tape. 

15.  Language  Laboratory  -  Audio  -  An  audio  presentational  device  that 
distributes  audio  information  via  a  control  console  to  student  stations 
equipped  with  headsets,  and  may  have  a  microphone  for  console/instructor- 
student  intercommunication  and  a  tape  recorder.  Student  may  interact 
with  taped  instructional  material,  rewind  and  play  back  or  store  responses. 
Student  responses  may  be  monitored  or  recorded  at  console. 

16.  Radio  System  -  AM/ FN  -  A  passive  audio  system  consisting  of  a 
broadcast  studio,  transmitting  station,  and  student  radio  receivers. 

The  system  uses  designated  AM/FM  frequency  bands  for  information 
transmission. 

17.  Radio  System  with  Responders  -  A  multi-channel  two-way  radio 
communication  system  that  operates  within  UHF  or  VHF-FM  frequency  bands 
limiting  broadcast  ranges.  Network  may  be  open  or  use  encoding/ 
decoding  techniques  or  responders  for  individual  channel  privacy. 

18.  Telephone  System  -  A  telephone  system  with  switching  matrix 
capability  that  allows  multiple  station  two-way  audio  communication 
at  two  or  more  remote  locations. 


Audio-Visual  Systems 

19.  Carrel  -  A  small  enclosure  or  alcove  incorporating  a  desk,  can 
be  used  by  one  or  two  trainees  and  can  be  equipped  with  AV  equip¬ 
ment  or  tools  or  print  materials  or  a  combination  of  these. 

20.  Dial  Access  Information  Retrieval  System  -  Audio/Video  -  Dial 
access  information  retrieval  is  an  electronic  system  for  distri buti ng 
audio  and/or  visual  materials  and  programs  which  are  stored  in  a 
location  remote  from  where  they  are  dialed  and  received. 

21.  Filmstrip  Projection  System  with  Audio  -  A  sound  filmstrip 
projector  represents  a  family  of  audio-visual  devices  using  single 
frame  visual  filmstrips  with  sound  on  magnetic  tape  or  records. 

Visuals  and  sound  may  be  manually  or  automatically  synchronized. 

Commercial  equipment  options  include  front  or  rear  screen  projection, 
remote  and  stop  action  capability,  and  cartridge  loading  models. 

22.  Microform  with  Audio  -  Microimagery,  such  as  microfilm,  used  as  a 
medium  of  instruction  with  audio  tape  or  disc,  etc. 

23.  Motion  Picture  Projection  System  -  16MM  and  Super  8MM  Films  - 
A  motion  picture  projection  system  using  professionally  or  locally 
prepared  16mm  or  S-8mm  sound  motion  picture  films  for  training. 

Appropriate  16mm  or  S-8mm  projector  and  projection  screen  are 
included. 

24.  Sound  Slide  Projection  System  -  A  system  for  presenting  infor¬ 
ma  tiorT^yliieans_oT^rraua’io  tape  and  a  series  of  synchronized 
projected  visual  slides. 

25.  Teaching  Machine  -  Still  Visual/Audio  -  An  individualized  instruction 
system  composed  of  programmed  instruction  still  frames,  such  as  large 
step  multiple  choice  or  fixed  linear  sequence,  and/or  synchronized  sound, 
and  a  manually  controlled  device  to  display  the  audio  and/or  visual 
information. 

26.  Teaching  Machine  -  Motion  Visual/Audio  -  An  individualized  instruction 
system  composed  of  programmed  instruction  motion  frames,  such  as  large 
step  multiple  choice  or  fixed  linear  sequence,  and/or  synchronized 

sound,  and  a  manually  controlled  device  to  display  the  audio  and/or 
visual  information. 

27.  Televideo  System  -  A  telecommunication  system  that  allows  audio 
and  visual  two-way  communication  between  two  or  more  remote  locations. 
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28.  Television  -  Cable  (CATV)  -  A  hybrid  CCTV  system  offering  selective, 
multiple  channel,  encoded  programming  to  cable  network  patrons.  A 
typical  system  consists  of  a  signal  receiving  antenna  system  for  the 
master  station  and  relay  of  amplified  signal  channels  via  area  substations 
to  system  subscribers.  Programming  may  also  be  generated  and  transmitted 
between  substations  offering  multiple  options  for  conference  or  training. 
Programs  are  encoded  for  privacy  and  control  of  viewing  audience. 

29.  Television  -  Cartridge  (CTV)  -  A  cartridge  television  system  (CTV) 
consists  of  packaged  video  tape  programs,  video  recorder,  playback 

and  display  units,  and  control  equipment  offering  high  selectivity 
and  availability  for  individualized  programming.  Program  cartridges 
may  be  prerecorded,  locally  produced,  or  recorded  off-the-air. 

30.  Television  -  Closed  Circuit  (CCTV)  -  CCTV  is  an  electronic  transmission 
system  for  images  and  sound  using  a  coaxial  cable  distribution  system. 
System  design  includes  one  or  more  studios  or  control  rooms,  a  signal 
distribution  center,  and  signal  distribution  cables  terminating  in 
reception  areas  equipped  with  receiver/monitors.  Off  air,  live  or 

video  taped  programs  may  be  used. 

31.  Television  -  Non-Magnetic  Video  Disc  System  -  An  experimental 

form  of  television,  similar  in  function  to  cartridge  television,  in  which 
the  program  is  encoded  on  a  thin  plastic  disc,  distributed  to  users  where 
it  is  rotated  at  high  revolutions  per  minute  on  a  player  which  reads 
the  data  and  sends  program  signals  into  the  antenna  terminals  of  a 
standard  color  television  receiver.  Random  access  capability. 

32.  Computer  -  A  programmable  electronic  device  that  can  store,  retrieve, 
and  process  data.  It  may  manage  and  display  information  to  a  student, 
accept  student  responses,  provide  feedback,  perform  calculations,  etc. 

33.  Demonstrator  -  A  low  fidelity  simulator  that  demonstrates  manipulative 
principles,  movement  in  time  or  space,  steps  of  a  procedure,  etc. 

34.  Mock-up  and  Panel  -  A  training  aid  used  to  demonstrate  relative 
shape,  size,  composition  or  function  of  an  object  or  system  display. 

It  may  have  moving  parts,  such  as  dials,  switches,  levers,  etc.,  or  have 
non-moving  parts,  such  as  cutaway  or  layout  models,  etc. 

35.  Operational  Equipment  -  A  unit  of  or  the  entire  equipment  used  on 
the  job  for  training  purposes  where  extreme  fidelity  is  required. 

36.  Physiological  Trainers  (Hostile  Environment)  Auditory  -  training 
devices  designed  to  place  controlled  stress  on  the  human  hearing  system 
through  use  of  a  physiologically  and/or  psychologically  adverse  sound 
environment,  to  enable  a  trainee  to  learn  to  function  in  this  adverse 
environment. 
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37.  Physiological  Trainers  (Hostile  Environment)  Visual  -  A  training 
device  designed  to  place  controlled  stress  on  the  human  visual  system, 
through  the  use  of  physiologically  and/or  psychologically  adverse  or 
low  threshold  visual  signals,  to  enable  a  trainee  to  learn  to  function 
In  this  adverse  environment. 

38.  Physiological  Trainers  (Hostile  Environment)  Surface  and  Internal 
Senses  -  A  broad  category  of  training  devices  designed  to  provide  the 
cutaneous,  kinesthetic  and  olfactory  sensors  with  physiologically  and/ 
or  psychologically  adverse  signals,  to  enable  a  trainee  to  function 

in  adverse  pressure,  temperature,  pain  or  disorientating  motion  environ¬ 
ments. 
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MEDIA  RELATED  VARIABLES 


STIMULI  CHARACTERISTICS 
Medium  of  Stimuli  Presentation 


1.  Visual  Cues  -  Signals  received  through  the  sense  of  sight. 

2.  Audio  Cues  -  Signals  received  through  the  sense  of  hearing. 

3.  Tactile  Cues  -  Signals  received  through  the  sense  of  touch,  including 
sensations  related  to  texture,  size,  shape,  or  vibration  of  the  skin. 

4.  External  Stimulus  Motion  Cues  -  The  sensations  felt  by  a  person  when 
he  is  moved  by  some  outside  force  in  such  a  way  that  his  body  experiences 
roll,  pitch,  yaw,  heave,  sway  and/or  surge. 

5.  Internal  Stimulus  Motion  Cues  -  The  sensations  felt  by  a  person  when 
he  moves  his  arm,  leg,  fingers,  etc. 

6.  Olfactile  Cues  -  Signals  received  through  the  sense  of  smell. 

7.  Gustatile  Cues  -  Signals  received  through  the  sense  of  taste. 

Visual  Form 

8.  Visual  Alphanumeric  -  Words  and/or  numbers  presented  visually. 

9.  Visual  Symbolic  -  Symbols  presented  graphically. 

10.  Visual  Graphic  -  Two-dimensional  figures,  such  as  maps,  graphs, 
mathematical  curves,  etc.,  presented  visually. 

11.  Visual  Pictorial  -  Two-dimensional  images,  such  as  photographs, 
drawings,  etc.,  presented  visually. 

12.  Visual  Solid  Object  -  A  three-dimensional  image  or  reality  that  is 
viewed  from  exterior  perspectives. 

13.  Visual  Environment  -  A  three-dimensional  image  or  reality  that  is 
v i ewed  from  inside. 

Visual  Movement 

14.  Visual  Still  -  A  static  visual  field,  as  with  a  still  photograph, 
drawing  or  printed  page. 

15.  Visual  Limited  Movement  -  A  basically  static  visual  field  with  elements 
that  can  be  made  to  move,  as  with  an  animated  transparency  or  simple  panel 
with  switches  that  move. 
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16.  Visual  Full  Movement  -  A  visual  field  in  which  all  elements  can  move, 
as  with  a  motion  picture,  flight  simulator,  or  operational  aircraft. 

17.  Visual  Cycl ic  Movement  -  A  visual  field  which  moves  through  a  fixed 
sequence  and  then  repeats  the  sequence  in  a  repetitive  manner,  as  with  a 
film  loop. 

Visual  Spectrum 

18.  Black  and  White  -  A  visual  field  composed  of  either  black  or  white 
elements,  as  with  the  printed  page  or  line  drawings. 

19.  Gray  Scale  -  A  visual  field  composed  of  black,  white  and  continuous 
gradations  or  gray,  as  with  a  black  and  white  photograph  or  television 
picture. 

20.  Color  -  A  visual  vield  composed  of  various  segments  of  the  visual 
spectrum,  as  with  color  television  or  motion  pictures. 

Visual  Scale 

21.  Exact  Scale  -  Actual  visual  field  or  a  one-to-one  replication  of 
that  field  as  with  a  full-sized  mock-up,  simulator,  or  operational  system. 

22.  Proportional  Scale  -  A  representation  of  reality  in  other  than 
full  scale,  such  as  a  scaled  model  map  or  photograph. 

Visual  Contrast 

23.  Dim  -  A  visual  object  which  blends  in  with  its  background,  i.e., 
there  is  a  small  or  no  luminance  difference  between  an  object  and  its 
background. 

24.  Bright  -  A  visual  object  which  is  brighter  than  its  background, 
i.e.,  there  is  a  large  luminance  difference  between  an  object  and 
its  background. 

Audio  Sources 

25.  Tonal  Sound  -  A  very  limited  source  of  sound  or  noise  which  is  used, 
rather  than  speech,  for  signaling  or  warning,  e.g.,  horns,  whistles,  sirens, 
bells,  buzzers,  etc. 

26.  Voice  Sound  -  A  limited  source  of  sound  which  enables  spoken  words 
to  be  used  as  the  medium  of  communications,  but  not  suited  to  more 
demanding  tasks,  such  as  music  or  sound  recognition  exercises. 

27.  Full  Sound  -  A  source  of  sound  that  contains  all  the  significant 
elements  of  the  sound  and  is  suited  to  the  demanding  task  of  sound 
recognition  exercises. 
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28.  Ambient  Sound  -  A  complex  sound  environment  with  sounds  emanating 
from  various  sources  and  from  various  directions,  including  background 
noise  and  task  significant  sounds. 

Audio  Stimuli  Intensity 

29.  Weak  -  Audio  stimuli  presented  to  the  trainee  which  have  weak  intensity. 

30.  Strong  -  Audio  stimuli  presented  to  the  trainee  which  have  strong 
intensity. 

Stimuli  Presentation 

31.  Static  -  A  unitary  stimuli  situation,  i.e.,  stimuli  are  presented  to 
the  trainee  "all  at  once”,  e.g.,  batch  presentations. 

32.  Dynamic-Ordered  -  A  sequential  stimuli  situation,  i.e.,  stimuli  are 
presented  to  the  trainee  sequentially  or  in  an  ordered  manner  over  time. 

-33.  Dynamic-Random  -  Anon-sequential  stimuli  situation,  i.e.,  stimuli 
are  presented  to  the  trainee  randomly  over  time. 

Stimuli  Presentation  Rate 

34.  Slow  Rate  -  A  slow  rate  or  speed  of  presentation  of  stimuli  to  the 
trainee,  allowing  the  trainee  a  long  or  maximum  stimulus  analysis  time. 

35.  Fast  Rate  -  A  fast  rate  or  speed  of  presentation  of  stimuli  to  the 
trainee,  allowing  the  trainee  a  short  or  minimum  stimulus  analysis  time. 

Number  of  Channels  or  Sources 


36.  Limited  -  A  small  number  of  sources,  channels,  or  instruments 
through  which  stimuli  are  presented  to  the  trainee. 

37.  Unlimited  -  A  multiple  number  of  sources,  channels,  or  instruments 
through  which  stimuli  are  presented  to  the  trainee. 

Stimuli  Distribution 

38.  Individual  -  All  information  is  presented  directly  to  one  individual 
trainee. 


39.  Group  -  Information  is  presented  to  a  group  of  trainees,  allowing  only 
indirect  access  to  the  information  for  an  individual. 
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. (COH'T)  MEDIA  SELECTION  MATRIX 


Audio  Only  V  -  Visual  Only 
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MEDIA  RELATED  VARIABLES 

RESPONSE  CHARACTERISTICS 
Response  Mode  of  Implementation 

1.  Overt  Response  -  Verbal  -  A  response  which  the  trainee  expresses  in  an 
audible  (verbal)  manner,  such  as  a  verbal  short  answer  response  to  a  question 
having  a  limited  set  of  correct  answers,  a  conversational  response,  or  a 
verbal  decision  response. 

2.  Overt  Response  -  Written  -  A  response  which  the  trainee  expresses 

in  an  observable  (written)  manner,  such  as  a  free  style  written  response, 
a  written  multiple  choice  response,  or  a  written  fill-in-the  blank  response. 

3.  Overt  Response  -  Manipulative  Acts  -  A  response  which  the  trainee  expresses 
in  an  observable  (manipulative)  manner,  such  as  the  small  movements  of  dials, 
swi tches ,  keys,  or  small  adjustments  to  instruments  or  the  large  movements  of 
levers,  wheels  or  use  of  hand  held  tools. 

4.  Overt  Response  -  Tracking  -  A  response  which  the  trainee  expresses  in  an 
observable  (tracking)  manner?  such  as  continuously  controlling  a  constantly 
changing  system,  e.g.,  steering  an  automobile. 

5.  Overt  Response  -  Procedural  Performance  -  A  response  which  the  trainee 
expresses  in  an  observable  (procedural  performance)  manner,  such  as 
performing  a  sequence  of  steps  in  a  procedure,  e.g.,  carrying  out  the  items 
on  the  checklist  for  preflighting  an  aircraft  or  turning  on  a  radar  system. 

Intensity  of  Response 

6.  Weak  -  Responses  made  by  the  trainee  with  weak  intensity. 

7.  Strong  -  Responses  made  by  the  trainee  with  strong  intensity. 

Response  Implementation 

8.  Static  -  A  unitary  response  situation,  i.e.»  responses  are  made  by  the 
trainee  "all  at  once". 

9.  Dynamic-Ordered  -  A  sequential  response  situation,  i.e.,  responses  are 
made  by  the  trainee  sequentially  or  in  an  ordered  manner  over  time. 

10.  Dynamic- Random  -  A  non-sequential  response  situation,  i.e.,  responses 
are  made  by  the  trainee  randomly  over  time. 


F\ 


Required  Response  Rate 

11.  Slow  Rate  -  A  slow  rate  or  speed  of  trainee  response,  l.e.,  a  rate  which 
allows  the  trainee  a  long  or  maximum  response  time. 

12.  Fast  Rate  -  A  fast  rate  or  speed  of  trainee  response,  l.e.,  a  rate  which 
allows  the  trainee  a  short  or  minimum  response  time. 

Number  of  Response  Channels 

13.  Limited  -  A  limited  number  of  sources,  channels,  or  instruments  through 
which  required  responses  are  made  by  the  trainee. 

14.  Unlimited  -  An  unlimited  number  of  sources,  channels,  or  instruments 
through  which  responses  are  made  by  the  trainee. 

Response  Distribution 

15.  Individual  -  One  individual  trainee  makes  the  required  response. 

16.  Group  -  A  group  of  trainees  make  the  required  response. 
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MEDIA  SELECTION  MATRIX 
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MEDIA  RtLATED  VARIABLES 


INFORMATION  FEEDBACK  LOGIC 
Medium  of  Feedback  Presentation 

1.  Visual  -  Feedback  presented  visually  by  means  of  a  display,  it  may  be 
coded  and  transmitted  visually  to  the  trainee. 

2.  Aural  -  Feedback  presented  aurally  by  means  of  a  display  to  the  trainee. 

3.  Written  Form  -  Feedback  presented  to  the  trainee  in  written  form. 

4.  Face- to- Face  Communication  -  Feedback  presented  by  direct  verbal  means 
to  the  trainee. 

5.  Indirect  Communication  -  Feedback  presented  by  indirect  verbal  means, 
such  as  by  intercom,  telephone,  or  radio  link. 

6.  Tactile  -  Feedback  presented  to  the  trainee  through  the  sense  of  touch, 
including  sensations  related  to  texture,  shape,  size,  or  vibration  of  the  skin. 

7.  Kinesthetic  -  Feedback  presented  to  the  trainee  by  either  internal  or 
external  bodily  movement,  such  as  reaching,  grasping,  tilting,  etc. 

8.  Olfactile  -  Feedback  presented  to  the  trainee  through  the  sense  of  smell. 

9.  Gustati le  -  Feedback  presented  to  the  trainee  through  the  sense  of  taste. 
Source  of  Feedback 

10.  Intrinsic  F  -  Information  or  cues  built  into  the  system  from  which  the 
trainee  interprets  feedback  information. 

11.  Extrinsic  F  -  Information  or  cues  not  inherent  in  the  trainee  action  or 
system  operations  but  is  supplied  by  an  external  source. 

Type  of  Feedback 

12.  Research  Correctness  (Rcr)  -  Information  about  the  correctness  or 
incorrectness  of  trainee's  response,  when  several  response  alternatives  are 
possible  and  the  correct  choice  is  not  known  to  the  trainee  in  advance. 

(Also  known  as  augmented  feedback.) 

13.  Response  Correctness  (Ref)  -  Information  provided  to  the  trainee  (or 
others  v/ho  need  to  know  about  his  performance)  that  he  has  in  fact  performed 
an  operation,  but  does  not  say  anything  about  the  longer  range  consequences 
of  the  action  taken. 
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14.  Response  Consequences  (Ren)  -  Information  about  the  consequences  of  the 
action  taken.  It  confirms  the  response  made  by  the  trainee,  and  the 
correctness  of  a  response  can  be  Inferred  only  from  its  consequences.  May 
also  serve  to  cue  the  trainee  to  perform  the  next  response  in  sequence. 

15.  System  Status  (Rss)  -  Information  about  the  condition  of  one’s  own 
or  another  system  or  the  external  environment,  on  the  basis  of  which  a 
trainee  or  team  must  act.  Information  is  not  necessarily  (or  even  frequently) 
the  immediate  consequence  of  or  follow-on  to  a  specific  trainee/team  action; 
it  may  reflect  system  events  that  have  been  put  in  motion  by  much  earlier 
trainee  actions.  Provides  information  that  regulates  trainee  and  system 
actions  in  the  sense  that  when  a  particular  status  condition  occurs,  the 
trainee  must  often  take  action  to  maintain  the  integrity  of  his  system. 

Feedback  Distribution 

16.  Individual  -  Feedback  is  presented  to  one  individual  trainee. 

17.  Group  -  Feedback  is  presented  to  a  group  of  trainees,  allowing  only 
indirect  access  for  an  individual. 


TRAINING  EQUIPMENT  &  MATERIALS 


51 


APPENDIX  B 


METHOD 


1.  Descriptions  of  Training  Methods  (B-2-24) 

2.  Functional  Context  Variables  (B -25-26) 

3.  Method  Selection  Matrix  (B-27) 


DESCRIPTIONS  OF  TRAINING  METHODS— CODING  EXPLANATIONS 

1.  Who  is  Performing? 

a.  ^Instructor 
S=Student 
M=Media 

b.  Ip  Sp  or  M^First  individual  or  group 
I 2*  S2*  or  M2”^econ^  individual  or  group 
Etc  • 

c.  (N+) “Expected  number  of  individuals  or  more  . 

(N-)=Expected  number  of  individuals  or  less  ^  ~  um*>er  ceil) 

2.  How  Many  Times  is  He  Performing? 

a.  l=Once 
2”Twice 
Etc. 

N=>Number  of  times 

3.  Where  is  His  Performance  in  the  Sequence? 

a.  l“First 
2«Second 

l,3=First  and  third 
Etc. 

.... “Cyclical  sequence 
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DESCRIPTION:  a  semiformal  discourse  in  which  two  (or  more)  instructors  present  a  series  of  events, 
facts,  concepts,  or  principles;  explore  a  problem;  or  explain  relationships.  Trainees 
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instructor  actually  performs  an  operation  or  does  a. job,  thereby  showing  the  trainee 
t  to  do  and  how  to  do  it;  he  then  uses  explanations  to  point  out  why,  where,  and  when 
is  done.  Usually,  the  trainee  is  expected  to  be  able  to  repeat  the  job' or  operation 
er  the  demonstration. 


MOW  LONG  I 
PERFORMANCE? 


DESCRIPTION:  A  method  In  which  the  trainee  Is  required  to  perform  under  controlled  conditions,  the  oper¬ 
ation,  skill-,  or  movement  being  taught.  Performance  Is  learning  by  doing.  In  the  student  tutor  method, 
trainees  are  paired  and  members  of  each  pair  perform  alternately  as  Instructor  and  trainee. 


HOW  LONO  IS  HIS 
PERFORMANCE?  (*  TIME) 


DESCRIPTION:  A  method  In  which  the  trainee  is  required  to  perform  under  controlled  conditions,  the  opera 
tion,  skill or  movement  being  taught.  Performance  is  learning  by  doing.  In  group  controlled  practice, 
trainees  work  together  at  the  rate  set  by  the  instructor,  step  by  step  and  "by  the  numbers" . 


MOW  LONG  I 
PERFORMANCE? 


DESCRIPTION:  A  method  of  self-instruction  in  which  the  trainee  works  through  a  carefully  sequenced  and 
pretested  series  of  steps  leading  to  the  acquisition  of  knowledge  or  skills  representing 
the  instructional  objectives.  The  trainee  proceeds  through  the  program  at  his  own  rate, 
responds  actively  (or  covertly)  to  each  step  in  the  sequence,  and  receives  immediate 


HOW  LONO  IS  HIS 
PERFORMANCE?  (X  TIME) 
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HOW  LONG  I 
PERFORMANCE? 
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DESCRIPTION:  Involves  in-depth  group  discussion  .of  real-life  situations.  It  requires  reading,  study, 
analysis,  discussion,  and  free  exchange  of  ideas  as  well  as  decision  making  and  the  selling  of  decisions 
to  others.  A  case  report  is  distributed  to  trainees.  The  report  contains  a  factual  and:  accurate 


WHAT  IS  HIS  DEGREE 
OF  CONTROL?  (*) 


HOW  LONG  IS  HIS 
PERFORMANCE?  (%  TIME) 


HOW  LONG  IS  HIS 
PERFORMANCE?  {%  TIME) 


HOW  LONG  IS  HIS 
PERFORMANCE?  (X  TIME) 


DESCRIPTION:  Games  include  a  set  of  structured  decision-making  tasks  typical  of  a  real-life  situation 

and  provide  a  systematic  means  of  observing  and  evaluating  trainees'  decisions.  These,  then,  are  fed 
back  to  the  trainees  so  that  they  can  judge  their  appropriateness.  Most  games  are  played  by  one  or 
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WHAT  IS  HIS  DECREE 
OF  CONTROL?  [%) 


DESCRIPTION:  The  in-basket  situation  is  composed  of  a  representative  sample  of  a  full  year’s  performance 
in  all  aspects  of  a  job.  Trainees  are  given  background  materials,  organization  charts,  policy  manuals, 
financial  statements,  reports,  and  position  papers  to  study  before  the  exercise  begins.  'Each  trainee  is 
then  exposed  to  a  structured  array  of  memos,  reports,  letters,  telephone  calls,  visits,  and  meetings.  In 


HOW  LONG  IS  HIS 
PERFORMANCE?  (%  TIME) 


WHAT  IS  HIS  DEGREE 
OF  CONTROL?  {%) 


DESCRIPTION:  A  group  of  trainees,  ranging  in  size  from  three  to  seven,  is  given  a  special  assignment 
in  the  form  of  a  problem.  The  group  is  asked  to  investigate  the  problem,  reach  conclusions,  and 
recommend  a  solution  or  a  course  of  action.  The  committee  may  produce  a  report  which  is  often  presented 
orally  to  the  larger  group. 


HOW  LONG 
PERFORMANCE? 


DESCRIPTION:  A  carefully  planned  visit  or  tour  to  a  place  away  from  the  training  activity.  The  purpose 

of  the  trip  is  to  provide  firsthand  observation  of  objects,  processes,  operations,  and  situations  not 
transportable  to,  or  reproducible  in,  the  training  facility.  The  field  trip  may  take  less  than  an  hour 


FUNCTIONAL  CONTEXT  VARIABLES 


ROLE  OF  ELEMENT  -  The  social  function  performed  by  the  trainee  within  the 
system's  operational  context. 

1.  Supervisory  -  The  trainee's  function  is  unequal  to  functions  being 
performed  by  other  individuals;  the  role  is  basically  one  of  overseeing  or 
directing. 

2.  Team  Performance  -  The  trainee's  function  is  equal  to  functions  being 
performed  by  other  individuals;  the  function  is  basically  one  of  teamwork 
and  cooperation. 

3.  Individual  Performance  -  The  trainee's  function  is  one  of  performing 
alone,  usually  without  supervision  or  team  assistance. 

FUNCTION  PERFORMED  IN  ROLE  -  The  primary  actions  performed  within  each  role. 

4.  Mental  -  An  action  occurring  or  experienced  in  the  trainee's  mind,  as 
contrasted  with  overt  physical  activity. 

5.  Physical  -  An  overt  bodily  action  performed  by  the  trainee. 

6.  Perceptual  -  An  action  by  the  trainee  involving  perception  or  observation. 

7.  Communicative  -  An  action  by  the  trainee  in  which  he  transmits  either  a 
written  or  verbal  message. 

STABILITY  OF  FUNCTION  -  The  function's  state,  quality,  or  degree  of  being 
constant  overtime. 

8.  Unstable  -  The  trainee's  function  is  not  constant  or  regular,  it  is 
characterized  by  continual  change  and  fluctuation. 

9.  Stable  -  The  trainee's  function  has  little  change  or  fluctuation  over  time. 

i PHYSICAL  CONTEXT  -  The  significance  of  the  physical  environment  upon 
performance  of  the  task. 

10.  Low  Impact  -  The  physical  environment  has  little  or  no  significant 
impact  on  performance  of  the  task. 

11.  High  Impact  -  The  physical  environment  has  a  large  or  significant  impact 
on  performance  of  the  task. 

PSYCHOLOGICAL  IMPACT  -  The  significance  of  the  psychological  environment 
upon  performance  of  the  task. 
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12.  Low  Impact  -  The  psychological  environment  has  little  or  no  significant 
impact  on  performance  of  the  task. 

13.  High  Impact  -  The  psychological  environment  has  a  large  or  significant 
impact  on  performance  of  the  task. 


METHOD  SELECTION  MATRIX 


APPENDIX  C 


I 

future  research  resource  variables 

1.  Method  Related  Variables 

a.  Stimuli  Characteristics  (c-2-3) 

b.  Response  Characteristics  (c -4-5) 

c.  Information  Feedback  Logic  (C-6) 

2.  Content  Related  Variables 

a.  Stimuli  Characteristics  (C-7-8) 

b.  Response  Characteristics  (C-9-10) 

c.  Information  Feedback  Logic  (C -11-12) 

3.  Information  Structure  Related  Variables 

a.  Stimuli  Characteristics  (C -13-14) 

b.  Response  Characteristics  (C -15-16) 

c.  Information  Feedback  Logic  (C -17-18) 
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METHOD  RELATED  VARIABLES 


STIMULI  CHARACTERISTICS 


Medium  of  Stimuli  Presentation 

1.  Visual  Cues  -  Signals  received  through  the  sense  of  sight. 

2.  Audio  Cues  -  Signals  received  through  the  sense  of  hearing. 

3.  Tactile  Cues  *  Signals  received  through  the  sense  of  touch,  including 
sensations  related  to  texture,  size,  shape,  or  vibration  of  the  skin. 

4.  External  Stimulus  Motivation  Cues  -  The  sensations  felt  by  a  person 
when  he  is  moved  by  some  outside  force  in  such  a  way  that  his  body 
experiences  roll,  pitch,  yaw,  heave,  sway  and/or  surge. 

5.  Internal  Stimulus  Motion  Cues  -  The  sensations  felt  by  a  person  when 
he  moves  his  arm,  leg,  fingers,  etc. 

6.  01  tactile  Cues  -  Signals  received  through  the  sense  of  smell. 

7.  Gustatile  Cues  -  Signals  received  through  the  sense  of  taste. 

Number  of  Channels  or  Sources 

8.  Limited  -  A  small  number  of  sources,  channels,  or  instruments  through 
which  stimuli  are  presented  to  the  trainee. 

9.  Unlimited  -  A  multiple  number  of  sources,  channels,  or  instruments 
through  which  stimuli  are  presented  to  the  trainee. 

Stimuli  Pacing 

10.  Self-paced  -  Stimuli  are  only  presented  upon  the  trainee's  request. 

11.  Forced-pace  -  Stimuli  are  presented  at  some  predetermined  rate, 
frequency  of  change,  etc. 

Stimuli  Distribution 

12.  Individual  -  All  information  is  presented  or  displayed  directly  to 
one  individual  trainee. 

13.  Group  -  Information  is  presentedor  displayed  to  a  group  of  trainees, 
allowing  only  indirect  access  to  the  information  for  an  individual. 
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Response  Criterion 

14.  Determinate  -  The  response  criterion  has  a  fairly  explicit  or  implicit 
response  requirement  as  a  direct  reflection  of  the  training  or  system 
requirement,  and  makes  the  stimulus  (visual,  audio,  etc.)  possess  a  certain 
or  determinate  value  or  payoff  for  the  trainee. 

15.  Indeterminate  -  The  response  criterion  is  highly  dependent  on  the 
trainee's  interpretation  of  the  response  requirement,  and  makes  the  stimulus 
(visual,  audio,  etc.)  possess  an  uncertain  or  indeterminate  value  or  payoff 
for  the  trainee. 


METHOD  RELATED  VARIABLES 


RESPONSE  CHARACTERISTICS 
Response  Mode  of  Implementation 

1.  Overt  Response  -  Verbal  -  A  response  v/hich  the  trainee  expresses  in  an 
audible  (verbal)  manner,  such  as  a  verbal  short  answer  response  to  a  question 
having  a  limited  set  of  correct  answers,  a  conversational  response,  ora 
verbal  decision  response. 

2.  Overt  Response  -  Written  -  A  response  which  the  trainee  expresses 

in  an  observable  (written)  manner,  such  as  a  free  style  written  response, 
a  written  multiple  choice  response,  or  a  v/ritten  fill-in-the  blank  response. 

3.  Overt  Response  -  Manipulative  Acts  -  A  response  which  the  trainee  expresses 
in  an  observable  (manipulative)  manner,  such  as  the  small  movements  of  dials, 
switches,  keys,  or  small  adjustments  to  instruments  or  the  large  movements  of 
levers,  wheels  or  use  of  hand  held  tools. 

4.  Overt  Response  -  Tracking  -  A  response  which  the  trainee  expresses  in  an 
observable  (tracking)  manner,  such  as  continously  controlling  a  constantly 
changing  system,  e.g. ,  steering  an  automobile. 

5.  Overt  Response  -  Procedural  Performance  -  A  response  which  the  trainee 
expresses  in  an  observable  (procedural  performance)  manner,  such  as 
performing  a  sequence  of  steps  in  a  procedure,  e.g.,  carrying  out  the 
items  on  the  checklist  for  preflighting  an  aircraft  or  turning  on  a  radar 
system. 

6.  Covert  Response  -  A  response  which  the  trainee  creates  in  his  mind  but 
does  not  express  in  an  observable  or  audible  manner. 

Response  Pacing 

7.  Self-paced  -  Responses  are  made  by  the  trainee  at  his  own  rate. 

8.  Forced-pace  -  Responses  are  made  by  the  trainee  at  some  .'redetermined 
rate  or  frequency . 

Response  Distribution 

9.  Individual  -  All  responses  are  expressed  by  one  individual  trainee. 

10.  Group  -  Responses  are  expressed  by  a  group  of  trainees,  allowing 
only  "indirect  responses  for  an  individual. 
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Response  Determinancy 

11.  Determinant  -  A  required  response  which  can  be  specified  in  advance 
of  the  stimuli  that  calls  the  response  out,  i.e.,  the  response  is 
preprogrammed  or  determined  before  the  operation  begins. 


12.  Indeterminant  -  A  required  response  which  cannot  be  specified  in 
advance  of  the  stimuli  that  calls  the  response  out,  i.e.,  the  response 
is  highly  dependent  on  the  immediate  stimuli  situation  and  cannot  be 
predicted  in  advance,  hence  is  indeterminate. 


— w— — 


METHOD  RELATED  VARIABLES 

INFORMATION  FEEDBACK  LOGIC 
Medium  of  Feedback  Presentation 

1.  Visual  -  Feedback  presented  visually  by  means  of  a  display,  it  may  be 
coded  and  transmitted  visually  to  the  trainee. 

2.  Aural  -  Feedback  presented  aurally  by  means  of  a  display  to  the  trainee. 

3.  Written  Form  -  Feedback  presented  to  the  trainee  in  written  form. 

4.  Face- to- Face  Communication  -  Feedback  presented  by  direct  verbal  means 
to  the  trainee. 

5.  Indirect  Communication  -  Feedback  presented  by  indirect  verbal  means, 
such  as  by  intercom,  telephone,  or  radio  link. 

6.  Tactile  -  Feedback  presented  to  the  trainee  through  the  sense  of  touch, 
including  sensations  related  to  texture,  shape,  size,  or  vibration  of  the  skin. 

7.  Kinesthetic  -  Feedback  presented  to  the  trainee  by  either  internal  or 
external  bodily  movement,  such  as  reaching,  grasping,  tilting,  etc. 

8.  01  tactile  -  Feedback  presented  to  the  trainee  through  the  sense  of  smell. 

9.  Gustatile  -  Feedback  presented  to  the  trainee  through  the  sense  of  taste. 
Source  of  Feedback 

10.  Intrinsic  F  -  Information  or  cues  built  into  the  system  from  which  the 
trainee  interprets  feedback  information. 

11.  Extrinsic  F  -  Information  or  cues  not  inherent  in  the  trainee  action  or 
system  operations  but  is  supplied  by  an  external  source. 

Feedback  Pacing 

12.  Self-paced  -  Feedback  presented  to  the  trainee  only  at  his  request. 

13-  Forced-pace  -  Feedback  presented  to  the  trainee  at  some  predetermined 
rate,  frequency,  etc. 

Feedback  Distribution 

14.  Individual  -  Feedback  is  presented  to  one  individual  trainee. 

15.  Group  -  Feedback  is  presented  to  a  group  of  trainees,  allowing  only  indirect 
access  for  an  individual. 
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CONTENT  RELATED  VARIABLES 


STIMULI  CHARACTERISTICS 
Medium  of  Stimuli  Presentation 


1.  Visual  Cues  -  Signals  received  through  the  sense  of  sight. 

2.  Audio  Cues  -  Signals  received  through  the  sense  of  hearing. 

3.  Tactile  Cues  -  Signals  received  through  the  sense  of  touch,  including 
sensations  related  to  texture,  size,  shape,  or  vibration  of  the  skin. 

4.  External  Stimul us  Motion  Cues  -  The  sensations  felt  by  a  person  when 
he  is  moved  by  some  outside  force  in  such  a  way  that  his  body  experiences 
roll,  pitch,  yaw,  heave,  sway  and/or  surge. 

5.  Internal  Stimulus  Motion  Cues  -  The  sensations  felt  by  a  person  when 
he  moves  his  arm,  leg,  fingers,  etc. 

6.  01  tactile  Cues  -  Signals  received  through  the  sense  of  smell. 

7.  Gustatile  Cues  -  Signals  received  through  the  sense  of  taste. 

Significance  of  Stimuli 

8.  Significant  -  The  stimuli  are  consequential  or  important  in  terms  of 
their  meaning  for  problem  solving,  e.g.,  the  identification  of  enemy 
bombers  or  missiles  may  direct  the  kind  of  weapons  released  against  them. 

9.  Insignificant  -  The  stimuli  are  inconsequential  or  unimportant  in  terms 
of  their  meaning  for  problem  solving,  e.g.,  identification  of  a  reconnaissance 
aircraft  may  be  less  significant  than  the  identification  of  an  enemy  bomber. 

Relevancy  of  Stimuli 


10.  Relevant  -  Accurate,  complete,  or  relevant  stimuli  which  the  trainee 
is  cued  to  respond  to  because  of  his  instructions  or  other  system  criteria. 

11.  Irrelevant  -  Inaccurate,  distorted,  incomplete  or  irrelevant  stimuli 
which  the  trainee  must  discriminate  from  relevant  stimuli,  e.g.,  "clutter" 
or  "noise". 

Stimuli  Cost 


12.  Low  -  A  low  cost  of  securing  information  (visual,  audio,  etc.  stimuli) 
to  help  make  a  decision  e.g.,  little  delay  or  effort,  or  loss  of  equipment 
or  men. 

13.  High  -  A  high  cost  of  securing  information  (visual,  audio,  etc.  stimuli 
to  help  make  a  decision,  e.g.,  much  delay  or  effort  or  loss  of  equipment 
and  men. 
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Stimuli  Similarity 

14.  Similarity  -  Stimuli  presented  to  the  trainee  which  resemble  other 
stimuTT! 

15.  Variability  -  Stimuli  presented  to  the  trainee  which  vary  or  differ 
from  other  stimuli. 
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CONTENT  RELATED  VARIABLES 


RESPONSE  CHARACTERISTICS 
Response  Mode  of  Implementation 

1.  Overt  Response  -  Verbal  -  A  response  which  the  trainee  expresses  In  an 
audible  (verbal)  manner,  such  as  a  verbal  short  answer  response-to  a  question 
having  a  limited  set  of  correct  answers,  a  conversational  response,  or  a 
verbal  decision  response. 

2.  Overt  Response  -  Written  -  A  response  which  the  trainee  expresses 

in  an  observable  (written)  manner,  such  as  a  free  style  written  response, 
a  written  multiple  choice  response,  or  a  written  fill-in-the  blank  response. 

3.  Overt  Response  -  Manipulative  Acts  -  A  response  which  the  trainee  expresses 
in  an  observable  (manipulative)  manner,  such  as  the  small  movements  of  dials, 
switches,  keys,  or  small  adjustments  to  instruments  or  the  large  movements  of 
levers,  wheels  or  use  of  hand  held  tools. 

4.  Overt  Response  -  Tracking  -  A  response  which  the  trainee  expresses  in  an 
observable  (tracking)  mannerT  such  as  continuously  controlling  a  constantly 
changing  system,  e.g. ,  steering  an  automobile. 

5.  Overt  Response  -  Procedural  Performance  -  A  response  which  the  trainee 
expresses  in  an  observable  (procedural  performance)  manner,  such  as 
performing  a  sequence  of  steps  in  a  procedure,  e.g.,  carrying  out  the  items 
on  the  checklist  for  preflighting  an  aircraft  or  turning  on  a  radar  system. 

6.  Covert  Response  ?  A  response  which  the  trainee  creates  ir.  his  mind  but 
does  not  express  in  an  observable  or  audible  manner. 

Response  Similarity 

7.  Similarity  -  Responses  made  by  the  trainee  which  resemble  other  responses. 

8.  Variability  -  Responses  made  by  the  trainee  which  vary  or  differ  from 
other  responses. 

Significance  of  Responses 

9.  Significant  -  The  responses  are  consequential  or  important  in  terms  of 
their  meaning  for  problem  solving,  e.g.,  pressing  a  "fire”  button  after 
identification  on  any  enemy  aircraft. 

10.  Insignificant  -  The  responses  are  Inconsequential  or  unimportant  in 
terms  of  Iheir  meaning  for  problem  solving,  e.g.,  social  communication 
among  team  members,  which  are  not  related  to  task  accomplishment. 
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Relevancy  of  Responses 

11.  Relevant  -  Accurate,  complete,  or  relevant  responses  which  the  trainee 
expresses,  according  to  his  instructions  or  other  system  criteria. 

12.  Irrel evant  -  Inaccurate,  distorted,  incomplete,  or  irrelevant  responses 
which  the  trainee  expresses,  e.g.,  a  response  which  gives  altitude 
information  only,  but  which  was  to  include  both  altitude  and  azimuth 
information. 

Response  Cost/Value  or  Payoff  Relationship 

13.  Low  Cost/Hiqh  Value  -  The  relationship  between  response  cost  and  value,  where 
the  cost  of  responding  a  particular  way  is  low  and  the  value  is  high,  e.g., 

a  decision  to  retreat  from  one  area  to  help  reinforce  another,  may  mean 
that  one  will  lose  some  men  and  equipment  in  the  area  from  which  they  are 
retreating,  but  will  save  more  men  and  equipment  in  the  long  run  by  reinforcing 
the  other  area. 


14.  High  Cost/Low  Value  -  The  relationship  between  response  cost  and 
value,  where  the  cost  of  responding  a  particular  way  is  high  and  the  value 
is  low  e.g.,  a  decision  to  delay  an  advance  may  cost  more  lives  and  loss 
of  equipment- than  a  decision  to  advance  immediately. 


CONTENT  RELATED  VARIABLES 


INFORMATION  FEEDBACK  LOGIC 
Medium  of  Feedback  Presentation 


1.  Visual  -  Feedback  presented  visually  by  means  of  a  display.  It  may  be 
coded  and  transmitted  visually  to  the  trainee. 

2.  Aural  -  Feedback  presented  aurally  by  means  of  a  display  to  the  trainee. 

3.  Written  Form  -  Feedback  presented  to  the  trainee  in  written  form. 

4.  Face-to-Face  Communication  -  Feedback  presented  by  direct  verbal  means 
to  the  trainee. 

5.  Indirect  Communication  -  Feedback  presented  by  indirect  verbal  means, 
such  as  by  Intercom,  telephone,  or  radio  link. 

6.  Tactile  -  Feedback  presented  to  the  trainee  through  the  sense  of  touch, 
including  sensations  related  to  texture,  shape,  size,  or  vibration  of  the  skin. 

7.  Kinesthetic  -  Feedback  presented  to  the  trainee  by  either  internal  or 
external  bodily  movement,  such  as  reaching,  grasping,  tilting,  etc. 

8.  Olfactile  -  Feedback  presented  to  the  trainee  through  the  sense  of  smell. 

9.  Gustatile  -  Feedback  presented  to  the  trainee  through  the  sense  of  taste. 

Type  of  Feedback 

10.  Research  Correctness  (Rcr)  -  Information  about  the  correctness  or 
incorrectness  ot  trainee's  response,  when  several  response  alternatives  are.- 
possible  and  the  correct  choice  is  not  known  to  the  trainee  in  advance. 

(Also  known  as  augmented  feedback.) 

11.  Response  Correctness  (Ref)  -  Information  provided  to  the  trainee  (or 

others  wno  need  to  know  about  his  performance)  that  he  has  in  fact  performed 

an  operation,  but  does  not  say  anything  about  the  longer  range  consequences 
of  the  action  taken. 

12.  Response  Consequences  (Ren)  -  Information  about  the  consequences  of  the 
action  taken.  It  confirms  the  response  made  by  the  trainee,  and  the 
correctness  of  a  response  can  be  inferred  only  from  its  consequences.  May 
also  serve  to  cue  the  trainee  to  perform  the  next  response  in  sequence. 


13.  System  Status  (Rss)  -  Information  about  the  condition  of  one's  own 
or  another  system  or  the  external  environment,  on  the  basis  of  which  a 
trainee  or  team  must  act.  Information  is  not  necessarily  (or  even  frequently) 
the  immediate  consequence  of  or  follow-on  to  a  specific  trainee/team  action; 

It  may  reflect  system  events  that  have  been  put  in  motion  by  much  earlier 
trainee  actions.  Provides  information  that  regulates  trainee  and  system 
actions  in  the  sense  that  when  a  particular  status  condition  occurs,  the 
trainee  must  often  take  action  to  maintain  the  integrity  of  his  system. 

Feedback  Specificity 

14.  Specific  -  Qualitative  -  Quanitative  (outcome)  information  provided 
to  the  trainee,  which  states  a  relationship  between  the  trainee's  response 
and  the  desired  correct  response;  for  example,  the  percent  error,  the 
number  of  correct  or  incorrect  responses,  or  rate  data. 

15.  Nonspecific  -  Qualitative  -  Qualitative  (outcome)  information  provided 
to  the  trainee  about  his  performance;  for  example,  "there  were  more  correct 
responses  than  last  time,"  "almost  all  ammunition  was  expended,"  etc. 

16.  Specific  -  Detailed  -  Detailed,  fractionated  categories  of  information 
or  a  wide  range  of  feedback  dimensions  provided  to  the  trainee  about  his 
performance. 

17.  Nonspecific  -  Nondetailed  -  Whole  unfractionated  categories  of 
information  or  a  narrow  range of  feedback  dimensions  provided  to  the  trainee 
about  his  performance. 

Properties  of  Feedback 


18.  Motivational  -  Feedback  provided  to  the  trainee  which  has  motivational 
(stimulating,  goal  adjusting,  and  criteria  selecting)  properties; 
motivational  feedback  has  few  or  no  cue  properties,  i.e.,  feedback  that  neither 
Informs  the  trainee  of  the  nature  or  locus  of  his  errors  nor  suggests 

how  these  might  be  corrected. 

19.  Directive  -  Feedback  provided  to  the  trainee  which  has  informational 

or  directive  properties;  directive  feedback  serves  a  guidance  function  to  the 
trainee  about  his  performance,  i.e.,  feedback  that  informs  the  trainee  of  the 
nature  and  locus  of  his  errors  and  suggests  how  these  might  be  corrected. 


INFORMATION  STRUCTURE  RELATED  VARIABLES 
STIMULI  CHARACTERISTICS 
Medium  of  Stimuli  Presentation 

1.  Visual  Cues  -  Signals  received  through  the  sense  of  sight. 

2.  Audio  Cues  -  Signals  received  through  the  sense  of  hearing. 

3.  Tactile  Cues  -  Signals  received  through  the  sense  of  touch,  including 
sensations  related  to  texture,  size,  shape,  or  vibration  of  the  skin. 

4.  External  Stimulus  Motion  Cues  -  The  sensations  felt  by  a  person  when 
he  is  moved  by  some  outside  force  in  such  a  way  that  his  body  experiences 
roll,  pitch,  yaw,  heave,  sway  and/or  surge. 

5.  Internal  Stimulus  Motion  Cues  -  The  sensations  felt  by  a  person  v/hen 
he  moves  his  arm,  leg,  fingers,  etc. 

6.  Olfactile  Cues  -  Signals  received  through  the  sense  of  smell. 

7.  Gustatile  Cues  -  Signals  received  through  the  sense  of  taste. 

Stimuli  Presentation 

8.  Static  -  A  unitary  stimuli  situation,  i.e.,  stimuli  are  presented  to 
the  trainee  "all  at  once",  e.g. ,  batch  presentations. 

9.  Dynamic-Ordered  -  A  sequential  stimuli  situation,  i.e.,  stimuli  are 
presented  to  the  trainee  sequentially  or  in  an  ordered  manner  over  time. 

10.  Dynamic- Random  -  A  non-sequential  stimuli  situation,  I.e.,  stimuli 
are  presented  to  the  trainee  randomly  over  time. 

Stimuli  Presentation  Rate 

11.  Slow  Rate  -  A  slow  rate  or  speed  of  presentation  of  stimuli  to  the 
trainee,  allowing  the  trainee  a  long  or  maximum  stimulus  analysis  time. 

12.  Fast  Rate  -  A  fast  rate  or  speed  of  presentation  of  stimuli  to  the 
trainee,  allowing  the  trainee  a  short  or  minimum  stimulus  analysis  time. 

Frequency  of  Stimuli  Change 

13.  Infrequent  -  A  low  frequency  of  stimuli  change,  i.e.,  stimuli  presented 
to  the  trainee  change  from  one  to  another  infrequently. 

14.  Frequent  -  A  high  frequency  of  stimuli  change,  i.e.,  stimuli  presented 
to  the  trainee  change  from  one  to  another  frequently. 
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Number  of  Stimuli 


15.  Few  -  A  small  number  of  separate  streams  of  signals  that  occur 
independently  of  each  other,  but  require  simultaneous  consideration. 

16.  Multiple  -  A  large  number  of  separate  streams  of  signals  that  occur 
independently  of  each  other,  but  require  simultaneous  consideration. 

Stimuli  Format 

17.  Formatted  -  Formatted  or  organized  stimuli,  such  as  alphanumeric 
displays. 

18.  Unformatted  -  Unformatted  or  unorganized  stimuli,  such  as  raw  video 
displays. 

Patterning  of  Stimuli 

19.  Simple  -  Simple  or  common  patterns  of  stimuli,  such  as  the  repetition 
of  the  same  event  at  high  frequencies  of  occurrence. 

20.  Complex  -  Complex,  detailed,  or  circumstantial  patterns  of  stimuli,  such 
as  discrete  sequenced  patterns  at  low  frequencies  of  occurrence. 


INFORMATION  STRUCTURE  RELATED  VARIABLES 


RESPONSE  CHARACTERISTICS 
Response  Mode  of  Implementation 


1.  Overt  Response  -  Verbal  -  A  response  which  the  trainee  expresses  in  an 
audible  (verbal)  manner,  such  as  a  verbal  short  answer  response  to  a  question 
having  a  limited  set  of  correct  answers,  a  conversational  response,  or  a 
verbal  decision  response. 

2.  Overt  Response  -  Written  -  A  response  which  the  trainee  expresses 

in  an  observable  (written)  manner,  such  as  a  free  style  written  response, 
a  written  multiple  choice  response,  or  a  written  fill-in-the  blank  response. 

3.  Overt  Response  -  Manipulative  Acts  -  A  response  v/hich  the  trainee  expresses 
in  an  observable  (manipulative)  manner,  such  as  the  small  movements  of  dials, 
switches,  keys,  or  small  adjustments  to  instruments  or  the  large  movements  of 
levers,  wheels  or  use  of  hand  held  tools. 

4.  Overt  Response  -Tracking  -  A  response  v/hich  the  trainee  expresses  in  an 
observable  (tracking)  manner,  such  as  continuously  controlling  a  constantly 
changing  system,  e.g.,  steering  an  automobile. 

5.  Overt  Response  -  Procedural  Performance  -  A  response  which  the  trainee 
expresses  in  an  observable  (procedural  performance)  manner,  such  as 
performing  a  sequence  of  steps  in  a  procedure ,  e.g.,  carrying  out  the  items 
on  the  checklist  for  preflighting  an  aircraft  or  turning  on  a  radar  system. 

6.  Covert  Response  -  A  response  which  the  trainee  creates  in  his  mind  but 
does  not  express  in  an  observable  or  audible  manner. 

Response  Implementation 

7.  Static  -  A  unitary  response  situation,  i.e.,  responses  are  made  by  the 
trainee  "all  at  once". 

8.  Dynamic-Ordered  -  A  sequential  response  situation,  i.e.,  responses  are 
made  by  the  trainee  sequentially  or  in  an  ordered  manner  over  time. 

9.  Dynamic-Random  -  Anon-sequential  response  situation,  i.e.,  responses 
are  made  by  the  trainee  randomly  over  time. 

Required  Response  Rate 

10.  Slow  Rate  -  A  slow  rate  or  speed  of  trainee  response,  i.e.,  a  rate 
which  allows  the  trainee  a  long  or  maximum  response  time. 

11.  Fast  Rate  -  A  fast  rate  or  speed  of  trainee  response,  i.e.,  a  rate 
which  allows  the  trainee  a  short  or  minimum  response  time. 
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Response  Frequency 

12.  Infrequent  -  A  low  frequency  of  response  change,  i.e.,  responses  made 
by  the  trainee  change  Infrequently. 

13.  Frequent  -  A  high  frequency  of  response  change,  i.e.,  responses  made 
by  the  trainee  change  frequently. 

Number  of  Required  Responses 

14.  Few  -  A  small  number  of  responses  that  the  trainee  is  required  to 
express,  each  independent  of  the  other. 

15.  Multiple  -  A  large  number  of  responses  that  the  trainee  is  required  to 
express,  each  independent  of  the  other. 

Response  Format 

16.  Formatted  -  Formatted  responses,  e.g.,  responses  to  a  programmed  text. 

17.  Unformatted  -  Unformatted  responses,  e.g.,  responses  made  by  team 
members  during  an  emergency  situation. 

Patterning  of  Responses 

18.  Simp! e  -  Simple  or  common  patterns  of  responses,  such  as  repetition  of 
the  same  response  at  high  frequencies  of  occurrence. 

19.  Complex  -  Complex,  detailed,  or  circumstantial  patterns  of  responses, 
such  as  discrete  sequenced  patterns  of  responses  at  low  frequencies  of 
occurrence. 
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INFORMATION  STRUCTURE  RELATED  VARIABLES 


INFORMATION  FEEDBACK  LOGIC 
Medium  of  Feedback  Presentation 


1.  Visual  -  Feedback  presented  visually  by  means  of  a  display,  it  may  be 
coded  and  "transmitted  visually  to  the  trainee. 

2.  Aural  -  Feedback  presented  aurally  by  means  of  a  display  to  the  trainee. 

3.  Written  Form  -  Feedback  presented  to  the  trainee  in  written  form. 

4.  Face-to-Face  Communication  -  Feedback  presented  by  direct  verbal  means 
to  the  trainee. 

5.  Indirect  Communication  -  Feedback  presented  by  indirect  verbal  means, 
such  as  by  intercom,  telephone,  or  radio  link. 

6.  Tactile  -  Feedback  presented  to  the  trainee  through  the  sense  of  touch, 
including  sensations  related  to  texture,  shape,  size,  or  vibration  of  the  skin. 

7.  Kinesthetic  -  Feedback  presented  to  the  trainee  by  either  internal  or 
external  bodily  movement,  such  as  reaching,  grasping,  tilting,  etc. 

8.  Olfactile  -  Feedback  presented  to  the  trainee  through  the  sense  of  smell. 

9.  Gustatile  -  Feedback  presented  to  the  trainee  through  the  sense  of  taste. 
Time  Schedule  for  Feedback 

10.  Pre-F  Interval  -  Immediate  -  Feedback  provided  immediately  after  the 
trainee's  antecedent  response,  i.e.,  there  is  a  small  or  no  interval  of  time 
between  the  antecedent  response  and  the  feedback  for  that  response. 

11.  Pre-F  Interval  -  Delayed  -  Feedback  provided  subsequently  after  the 
trainee's  antecedent  response,  i.e.,  there  is  an  interval  of  time  or 
delay  between  the  antecedent  response  and  the  feedback  for  tnat  response. 

12.  Post-F  Interval  -  Immediate  -  Presentation  of  the  next  stimulus 
immediately  after  the  occurrence  of  feedback  for  the  last  response,  i.e.’, 
there  is  a  small  or  no  interval  of  time  between  the  occurrence  of  feedback 
for  the  last  response  and  the  presentation  of  the  next  stimulus. 

13.  Post-F  Interval  -  Delayed  -  Presentation  of  the  next  stimulus  subsequently 

after  the  occurrence  of  feedback  for  the  last  response,  i.e.,  there  is  an 
interval  of  time  between  the  occurrence  of  feedback  for  the  last  response  and 
the  presentation  of  the  next  stimulus.  ~ 
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Feedback  Regularity 

14.  Regular  -  Feedback  provided  to  the  trainee  at  regular  intervals,  such  as 
after  every  or  every  other  response  or  at  established  or  fixed  periods. 

15.  Irregular  -  Feedback  provided  to  the  trainee  at  variable  intervals, 
which  may  change  as  a  function  of  stage  of  training  or  level  of  performance. 
This  includes  the  provision  for  intermittent  presentations  to  permit 
probabilistic  schedules  for  reinforcement. 

Frequency  of  Feedback 

16.  Frequent  -  Any  feedback  given  to  the  trainee  that  is  provided  at  least 
for  20-30%  of  all  his  responses. 

17.  Infrequent  -  Any  feedback  given  to  the  trainee  that  is  provided  less 
than  for  20%  of  all  his  responses  or  is  not  provided  at  all. 


Feedback  Format 

18.  Formatted  -  Formatted  feedback,  e.g.,  alphanumeric  feedback  displays. 

19.  Unformatted  -  Unformatted  feedback,  e.g.,  raw  video  feedback  displays. 
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APPENDIX  D 
SAMPLE  COST  SHEETS 

COST  DATA  COLLECTION  FORM 


Instructional  Delivery  System  _ 16P 

Run  ID  SAMPLE  SHEET  1 _ 


SELF  PACE 


SYMBOL 

VARIABLE  DESCRIPTION 

VALUE 

UNITS 

FACILITIES 

FACOST 

lotal  facilities  acquisition 
and/or  refurbishing^costs 

175,000 

Dollars 

LOFFA 

Expected  years  of  life  of 

Fflrnsr  asspfs  (Tn  whnlp  numhersl 

15 

WOm 

HS9HI 

Total  square  feet  required 
for  each  instructor 

64 

SQFTST 

otal  square  feet  required 
lier  student  position 

warn 

BHH 

SQFTAH 

Total  square  feet  required 
for  admiiii  strati ve  overhead 
for  ail  student„pttsitions - 

2,500 

Sq  Ft 

EQCISP 

Equip,  implementation  costs 
independent  of  stud.  pos. 

125.243 

Dollars 

LOFEQ1 

Expected  years  of  life  of 

FOCJSP  qssets 

10- 

f 

Equip,  implementation  costs 

_ 565 _ 

LOFEQ 

Expected  years  or  life  or 

FOIMPC  asspf.s  (in  whole  numbers! 

hubi 

TSPOSD 

Percent  of  operating  time 
student  nosition-.down - . — 

•01 

IRH9 
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SAMPLE  COSTING  SHEET  2 


SYMBOL 


VARIABLE  DESCRIPTION 


Instructional  Ma 


%  of  TLENGH  (i.e.,  time  spent 
in  training  medium)  for  which 
new  instructional  material 
must  be  developed 


U  of  original  development  cost 
required  each  year  to  maintain 
instructional  material 


%  of. original  development  cost 
remaining  at  end  of  planning 
Deriod 


Average  cost. of  developing 
one  hour  of  instructiona I 
material 


INISPO 


SALINR 


Instructor  to  student 
postion  ratio 


Annual  salary  and  benefits  of 
one  instructor 


SUPPLY 


Cost  of  expendable  supplies  for 
each  student  while  enrolled  in 


course 


STUDSL 


STCSTI 


STCST2 


Annual  salary  and  benefits  of 
one  student 


Average  student  travel  cost 


Average  per  student  travel  cost 


iscei laneous 


Number  of  years  in  planning  period 


Mis 


TLENGH 


m-liW: 


Average  time  spent  in  training 
medium  per  student 
(non-recvcled  students) 


Average  hours  per  week  student 


Average  time  the  recycled  student 
spends  repeating  material _ 


Percentage  of  excess  student 
positions  reguired  to  provide 
for  fluctuations  in  input 


Percent 


Percent 


Decimal 

Ratio 


1,168  {Dollars 


a 


5,656.  {Dollars 


Weeks 


Percent 


SAMPLE  COST  SHEET  3 


DISTRIBUTION 


1  US  AmHY  GtSItWN  COMMAND  Ml  IN!  APPt 

1  OEPAkTMEM  OF  I  he  NAVY  ThAINIuo  ANALYSIS  ANU  EVALUATION  GROUP 
l  H())A  ATTn:  OAAi,-£|t 

1  US  PmCIFIC  FLEET  HUMAN  HtSOURCt  MANA&tMF  .1  Ufc.TACHME.NT 
I  hJi  iCATa  ATTn:  aIlAT-uP-d 
P  HilOA  RtStARCH  an  j  STUUltb  OF  L 

1  MlLllARY  OCCUPATIONAL  UtVLLOPMENI  U 1 V  OAPC-MSP-0.  RM  R52C  HOFFMAN  BLOG  1 
4  OAsO  <MRA  AND  Li 
1  HQ  Tl  AT  A  TECHNICAL  LltiHAHY 
1  HU  )A  OUCSPEH 
I  HQUA  ATTN:  U AHI-1S1 

1  USA  AVIATION  S»ST£mS  COMO  All.v,:  UMSAV/i.H 


USA  lORAUCOM  at  In:  AMSlL-PA-hh 

heaowuarters  us  marine  corps  aiins  code  mtmt 
hEaquUARTerS*  u:,  marine  corps  a(TN:  cour  mPI-20 
US  Akmy  EUROPE  and  SEVlnTii  Army 

1ST  INFANTRY  DIVISION  ANU  r  T.  RILEY  A  T  Tr .  AF2N-UPT-T 

CH (Er ♦  A  T  T 1 TUuE  ♦  OPINION  SUHVtY  UlViSION  ATTN:  ATZI-NCR-MA,  HOFFMAN  BLOG  II 
USA  1 NTELL I GtNCr  ANO  SECURITY  lOMMANU  ATTN!  IA0PS*TNG-T 
HU  TkADOC  TECHi'ICAL  LIRRarY 

NAVAe  TRAINING  eQ.JIPMEnI  LEN  aIINI  TECHNICAL  LIBRARY 

MlLllARY  OCCUPATIONAL  ulVclOPmenI  UIhECTowATE  ATTN!  ATZl-NCR-MS-M,  Hm  JN33  hOFFmAn  BLUG  II 
DATA  ANALYSIS  UlViSION  AllNS  aIEI-NCR-MO,  HUFFMAN  OLDG  11 
USA  hILPERCEN  ATTN!  UuPC-POO-r 
RTH  iNFANlRY  DIVISION 

HQOA  ARMY  FORCE  mOUERNI^A I  ION  COORDINATION  OFFICE 

HUi)A  ATTN:  DASG-pTH 

1230  OSARCOM  RESERVE  CENltR 

OS  Army  SOLDIER  SJPPUR1  CENTER  / 

USA  rORCES  COhmanO  AFLG  -  OEPu I Y  CHIEF  OF  STAFF  FOR  LOGISTICS 

US  ArMY  AIR  DEFENSE  SCHOOL  ATTNS  ATSA-UTU 

OIHElTOHATF  OF  TRAINING  aTTN:  mUU-T 

OIRElTORATE  OF  COMBAT  UEVLLOPmENTS  ATTN:  AT2U*>D 

hqoarCOm  marine  CORPS  l I  a  I  Son  OFC 

DEPARTMENT  OF  TmE  army  OS  army  INTELLlGFNCE  ♦  SECORITY  COMMAND 
USA  MISSILE  command  ATTN:  UKSmI-NTN 
OrsEL- A  T  Du 


ofc  of  Equal  opportunity 

SOUTHERN  FLO  DIV 


US  AmMY  CECOM  aTTN 
USA  EURCES  command 
pm  Trade  / 

US  MiLITAHY  UISTRICT  OF  HAbhlNGlUN 
NAVAl  CIVILIAN  PERSONNEL  LUMO 
2P  ARl  LIAISON  OFF  ICE 

7Th  aHmy  training  commano 

AHMY  THAINING  SUPPORT  CENltR  INDIVIDUAL  TRAINING  EVAlUATlON 
hqda.  OCSDps  individual  iraInjng 
houa.  dcsops  training  uirecturate 
hqda «  ocslog  maintenance  management 
hqda.  DCS  SlUOY  office 

U.S.  NAVY  TRAINING  ANALYSIS  EVALUATION  GROUP 
USACutC  ATTN!  ATEC-EX-E  HUMAN  FACTORS 
ATTN;  SM-ALC/DPCR 
usaFmGOS/Tac  senior  army  auvisur 

INTEr-UNIV  SEMINAR  ON  ARMtU  FUHCtS  ♦  SOC 
UASA  (RUA)  UEPUlY  FUR  SCIENCE  ANU  TECHNOLOGY 
OFC  OF  NAVAL  RtSEAHCH  / 

AFHHL/LRT 
afhrl/LRLG 

AIR  >■  ORCE  HUMAN  RESOURCES  LAB 
AF  AMrL/GB 

NAVY  PERSONNEL  R  ANO  0  CENTER 
NAVY  PERSONNEL  P  ANO  0  CENTER 


ATTN:  AFhrl/TSR 

director  of  programs 
/ 


103 


i  OFC  uF  NAVAL  HLNEARCH  PEkSONNF  L  ANU  TRAINING  RESEARCH  PROGRAMS 
1  NAvAl  PERSONNEL  H  ♦  U  CtNTfcW  / 

1  OFC  oF  NAVAL  RLSEAHCH  PROJEC1  OFFICER.  ENVIRONMENTAL  PHYSIOLOGY 
1  MCF'AmN-gHAY  ♦  ASSOCIATES*  INC.  / 

1  NAvAl  AtPObPACt  M£U I  CAL  RSCH  LAO  AIRdOKNF  RANGER  RESFARCH 
]  OEPT.  OF  NAUONAL  DEFENCE  UEFF.NLt  AN  0  CIVIL  iNStITUTF  Of  FNVlP  mEp 
1  nAvAl  AEWOSPACt  MEDICAL  HOLM  lAo  AEROSPACE  PSYCHOLOGY  DEPAKImEnT 
i  os*  ihauoc  systems  analysis  activity  attn:  aTaa-tca 
I  HEADQUARTERS*  CijAST  GUARD  CHli-F*  PSYCHOLOGICAL  HSCrt  up 
1  MSA  mESEAHCM  AN  i  TECHNOLOGY  LAw  / 

1  USA  MOBILITY  FVI<J  1  AMEN  I  H  AND  U  COMO  AtlN:  ORUME-TU  (SCHOOL) 

I  NlGrtl  VISION  lAh  ATTN:  URSEL-MV-SUU 
1  USA  iRAINlF'G  ROakO  AT(n:  ATT G-Alb-T A 
1  USA  mATERIEL  SYSTEMS  analysis  rtdlVITY  ATTN:  URXSY-C 
1  NAFEc  HUMAN  ENGINEERING  BRANCH 

1  HA  T T c LLE-COLUMb' iS  LAFiOhaIukIES  TACTICAL  TECHNICAL  OFC 
)  USA  -HCTIC  TEST  CEN  attn:  AMSTE-PL-TS 
I  USA  cOLO  KFGlON-5  (ESI  CEN  ATTh!  S1ECH-OP 
1  DEFENSE  language  INSTITUTE  FOREIGN  LANGUAGE  CEN 
1  hu  w«AIH  OIV  UF  nEOROPSYLH 1 A  T w Y 

I  USa  cLECTHONiC  vAmFAKE  LAB  CHLtF,  InTEll IGENCE  MaTEr  OEVEl  ♦  SUPP  OFF 
1  US-.  kSCH  DEVEL  ♦  STANUARUIZA  GP*  U.K. 

i  usa  xEseapch  ani,  development  labs  Chief,  rehav  sciencfs  uiv.  food  sli  la 
1  naval  aim  systems  commanO  ATTn:  AIR-Sdn 
1  USACuEC  TFCHNlCAv.  INFORMATION  CtNTER 
I  USAAhL  LIHpAhy 

1  HOmAn  RESOURCES  PSCH  ORo  (HUMRRO)  / 

1  SEVILLE  RESEARCH  CORPORA! ION 

1  USA  IRAUOC  SYSTe.NS  ANALYSIS  ACTIVITY  ATTN,  AlAA-SL  (TtCH  LlBPARY) 

I  UN IFuRMEL)  SEHVlCES  UNIT  OF  ThE  HEALTH  SCI  OEPARTmENT  OF  PSYCHIATRY 
1  hOFAn  RESOURCES  RSCh  Org  IhUMRPU) 

1  HUMRhO  LlHRARY 
)  HATTeLLE  REPOHIS  LIBRARY 
1  RAN0  CORPORATION  AT  IN:  LIBRARY  U 
1  GRONlNGER  LIBRARY  attn:  ATZF-rS-L  BLDG  1313 
1  CEJTeR  FOR  NAvAL  ANALYSIS 
1  NAVAL  HEALTH  RSCH  CEN  LIBRARY 

1  NAVAL  PERSONNEL  R  ANU  0  ClN  LIrRARY  ATTn:  COOE  Pl<)6 
1  AIR  eORCE  human  RESOURCES  LAB  aTTN:  AFhhL/OTS 
1  MO,  eT.  huachuca  attn:  tech  rf.f  UIv 

1  USa  ACADEMY  of  HtALTH  SCIENCES  ST IMSON  IBRARY  (DOCUMENTS) 

1  SCHOOL  OF  SYSTEMS  ANU  LOGISTICS  / 

1  FRIC  PROCESSING  ANU  REFERENCE  FAC  ACQUISITIONS  LIBRARIAN 
1  DEPARTMENT  OF  The  NAVY  THAINING  ANALYSIS  and  EVALUATION  gp 
(  NATIONAL  CENTER  FOR  HEALTh  STATISTICS  / 

1  OSMA  OEP1  OF  BEHAVIORAL  SCI  AND  LEADERSHIP 
1  OLD  oOMINION  ONIVEHSITY  PERFORMANCE  ASSESSMENT  LABORATORY 
1  USA  COMMAND  AnU  GENERAL  SIAFF  COLLEGE  ATTN!  LIBRARY 
1  USA  IRANSPOHTATION  SCHOOL  USA  lHANSP  TECH  InFO  AnD  RSCH  LEN 
1  NMRUl  PROGRAM  MANAGER  FOR  HUmaN  PERFORMANCE 

i  usa  aomincfn  Technical  research  branch  library 
)  USA  fIELU  arty  ho  / 

1  NAT  CLEARINGHOUSE  FOR  menial  health  info  PAHKLABN  BLDG 
1  U  OF  TEXAS  CEN  FOR  COMMUNICATION  RSCH 
1  INSTITUTE  FOH  DEFENSE  ANALYSES 
i  usa  i Raining  support  center  attn:  atic-dst-pa 
I  AFHRl  TECHNOLOGY  OFC  (H) 

1  PUROoE  UNI V  t)EPT  OF  PSYCHOLOGICAL  SClENCtS 
l  USA  MOBILITY  EQUIPMENT  R  «NO  U  COMMaNJ  ATTN:,  DRUME.-4G 

i  ua  ub  army  retraining  sue  resf.ahch  ♦  evaluation  division 
1  ESSF*  CORPORATION  / 

I  HAZElTINE  CORPORATION  / 


104 


USA  -tfiOMtalCAL  RESEARCH  CAB  SCIENTIFIC  INFOHMAl  ION  CENTER 
HAYTnEON  SEHVICt  COMPANY  SPtNCtH  LABORATORY 
HiJIAim  RESOURCE  HANAGtMtNI  CEN t  BAN  ulLGO 
OSAFm  UERI  OF  nt-t  sd  *  LtAUtpSHlp 

us  military  acauEmy  otpi.  or  history*  blug  boi 
t.lb'i  i NTELL I Gt  NCF  CEN  ANU  SCh  aITN:  SCHOOL  LIBRARY 
USA  INTELLIGENCE  CtN  AND  SCH  AHN:  ATSI-ijP 
MAHl.vt  CORPS  INSTITUIE 
NAVAl  SAFtTY  CEn  T  £H  / 

us  cuast  guard  ins  cen  aiin:  educational  svcs  oeeicer 

GACCmUOET  COLLtut  LEARNING  CtNlfcR 
uSAAvNC  and  et.  R  joker  aiin:  AlAU-ES 
US  AmMY  A VN  TNG  LlhRAHY  ATTN:  CHltE  lIhRARIaN 
uSa  m  1 R  DEFENSE  school  aiin:  oISA-OT 
usaaknc  attn:  aT/g-o 

US  MILITARY  AC  At  it  AY  DlHECIQR  ,)E  INST  1  T  0  T  IONAL  RSCH 
USAAuS-LIBRARY-DOCUMtNlS 

USA  SERGEANTS  MAJOR  ACADtMY  AT  IN:  LEARNING  RESOURCES  CENTER 

USA  INFANTRY  BOah’j  ATIN:  AT2b-1B-AE 

US  i  INTELLIGENCE  CEn  AND  SCH  aITn:  ATSI-DT-SFL 

USA  uRONANCE  CtN  ANU  SCH  AIIN:  ATSL-TO-TAC 

USA  mRMOR  SCHOOL  ATTN:  Al/K-Ju 

UStt  mRMOR  CENTtw  OIRtCTORATt  OF  COMhAI  UFVtLOPMtNTS 

NAVAL  POSTGHADOaIE  SCH  AIIN:  ilOOLtY  ANOx  LIBRARY  ICOuF  14P4) 

USA  I RANSPORI A  T  fUN  SCHUOC  DEPUTY  ASST.  COMMANDANT  EDUCA.  TECHNOLOGY 
USA  SIGNAL  SCHUuL  ANU  FT.  OOHuON  ATTN:  aTZH-ET 
USA  hUAHTERMASIeR  SCH  ATTn:  AISM-TnG-IM-ET 
USA  MILITARY  POLICE  SCHOOL  attn:  LIBRARY 

USa  mRMOR  CF-NTtR  .FT.  KNOX  OEtiCt  OF  ARMOR  FORCE  MGT  ♦  STANUARDl/Al  ION 
CHIE>  OF  NAVAL  EOJCAIION  ahO  TnG  / 

USA  aIGNAL  SCHOOL  ♦  FT.  GuKUON  tOOCAUONAL  TECHNOLOGY  DIVISION 

HU  A  l  C/KPTlj  training  SYS  f  CMS  1/tvELOPNtNT 

USA  iNST  I  lUTE  F  ijh  MILI1AHY  ASSISTANCE  ATTN:  ATSU-TO-TA 

US  AmMY  ARMOR  CtNTtH  AITn:  AT/K-TD-PMO 

USA  «IR  UEFENSt  CENTtR.  FT.  BL I SS  ATTN;  ATZC-OIM 

USA  uUARTt  RMAS I  EH  SCHOOL  UlRtC  I  ORA  T  E  OF  TRAINING  OEVELORMENTS 

US  CuAST  GOARO  ACADtMY  / 

USA  I HANSPORT  A I  ION  SCHOOL  DIRECTORATE  OF  THAlNING  ♦  DOCTRINE 
USA  INFANTRY  SCHOOL  LIBRARY  / 

USA  INFANTRY  SCHOOL  ATTN:  ATSh-I-V 
OS  AmMY  INFANTRY  SCHOOL  ATTN:  AlSn-CJ 
USA  INFANTRY  SCHOOL  ATTN:  ATSh-UOT-LRO 
usa  infantry  school  attn:  atsh-ev 

USA  mP  ♦  CHEM  SCH/TNG  CEN  ♦  FT.  MCCLELLAN  ATTN:  ATZN-pTS 

USa  r*P  ♦  CHEM  SCH/TNG  CtN  ♦  FT.  MCCLElLAN  U1H.-  COMBAT  DEVELOPMENT 

USA  «P  ♦  CHEM  SCH/TNG  CEN  ♦  FT.  MCCLELLAN  DIR.  TRAINING  DEVELOPMENT 

USA  «P  ♦  CHEM  SCH/TNG  CtN  ♦  FT.  MCCLELLAN  ATTm:  ATZN-mP-ACE 

USA  INSTITUTE  OF  AUM InISTR AT IOn  ATTN:  RESIDENT  TRAINING  MANAGEMENT 

USA  FIELD  ARTILLERY  SCHOOL  MORRIS  s«ett  library 

USA  INSTITUTE  OF  ADMINISTRATION  ACADEMIC  LIBRARY 

USA  wAR  COLLtGt  ATTN:  LIBRARY 

USA  eNGINEER  SChOOL  LIBRARY  AnU  LEARNING  RESOURCES  CENTER 
USA  mRMOR  SCHOOl  (USARMS)  AT  In?  LIBRARY 
US  CuAST  GUARD  ACADEMY  LIBRARY 

USA  I HANSPORT  A  I  ION  SCHOOL  TRANSPORTATION  SCHOOL  LIBRARY 
ORGANIZATIONAL  EFPECTIVtNtSS  CfN  .  SCH  ATTN:  LIBRARIAN 
US  AmMY  INTELLIGENCE  CtNTtR  *  SCHOOL  ATTN:  ATSI-TR 

US  AmMY  INTELLIGENCE  CtNTtH  ♦  SCHOOL  ATTN:  ATSI-RM-M 

US  AmMY  INTELLIGENCE  CtNTtR  ♦  SCHOOL  ATTN:  ATJI-ES 

us  marine  corps  education  center 

USA  FIELD  ARTILLERY  SCHOOL  UiPtCTORATt  OF  COURSE  DEV  »  TRAINING 
OEPAmTMEnT  of  The  AIR  FORCE  AIR  UNIvEHSITY  LIBRARY  (ATC1 


105 


1 

I 

l 

1 

I 

if 

? 

1 

I 

1 

T 

1 

1 

P 

1 

h 

1 

1  4  0 
1 
1 
1 

A 

1 


USA  CHAPLAIN  CEnIlR  ♦  SCHOOL  AlIN:  ATSC-TD-00 

USA  CHAPLAIN  CENTER  ♦  SCHOOL  ATTN!  AfSC-IO-fcO 

USA  CHAPLAIN  CtNl-H  ♦  SCHOOL  AlIN!  ATSC-TD-SF 

USA  cHAPLAIN  CENlEK  ♦  SCHOOL  AlINS  A 1 SC-I 'OS-LLC 

HU  ThAUOC  TFIAInIVO  OEvfcLOPMtNT  iNSTITUfE 
HRITiSH  EMBASSY  ShITISfi  ulFENCL  STAFF 
CAuAol AN  JOINT  STAFF 
COLS  (A)  LIHHAHr 
F  RENCH  ARMY  A I T  ACHt 

AUsfnlAN  embassy  oefense,  military  ayu  air  aitaChE 

CANADIAN  OEFENCt  LIAISON  STAFF  ATlN:  COUNSELLOR*  DEFENCE  R  AND  U 

ROYAl  NETHEHLANuS  EMBASSY  MILITARY  ATIachE 

CAnAuIAN  F ORCES  BASE  CORNaaLLIs  aITns  PF  RSOnNEL  selection 

CANAuIAN  forces  PERSONNEL  APPL  rsch  UNIT 

ARMY  PERSONNEL  research  tsTABL  ISFIMtNT 

LIHRmRY  OF  COnorESS  EXCHANGE  and  OIF T  o|v 

defense  rtcHNiLAL  information  ctN  atin;  utIc-ooa-p 

LIBR-RY  OF  C0NORLSS  UNI!  uOCOMtNTS  EXPEuITlNO  PROJECT 

us  government  printing  ofc  library,  public  documents  department 
us  government  printing  ofc  library  ayu  statutory,  cIb  oiv  <scu 
THE  «RMy  CImRARY  ATIN:  ARMY  SlUDltS  StC 
/  / 

/  / 


number  uP  AOORESSttS  Put! 

total  number  of  copies  j«/ 


E 


16 


